Method and apparatus of encoding/decoding image data based on tree structure-based block division

ABSTRACT

Disclosed are methods and apparatuses for image data encoding/decoding. A method of decoding an image includes receiving a bitstream in which the image is encoded; obtaining index information for specifying a block division type of a current block in the image; and determining the block division type of the current block from a candidate group pre-defined in the decoding apparatus. The candidate group includes a plurality of candidate division types, including at least one of a non-division, a first quad-division, a second quad-division, a binary-division or a triple-division. The method also includes dividing the current block into a plurality of sub-blocks; and decoding each of the sub-blocks with reference to syntax information obtained from the bitstream.

RELATED APPLICATIONS

This application is a Continuation application of U.S. patentapplication Ser. No. 18/095,665, filed on Jan. 11, 2023, which is aContinuation application of U.S. patent application Ser. No. 17/580,053,filed on Jan. 20, 2022, which is a Continuation application of U.S.patent application Ser. No. 17/027,189, filed on Sep. 21, 2020, which isa Continuation application of U.S. patent application Ser. No.16/375,748, filed Apr. 4, 2019, now abandoned, which is a Continuationapplication of the International Patent Application Serial No.PCT/KR2017/011152, filed Oct. 10, 2017, which claims priority to theKorean Patent Application Serial No. 10-2016-0127890, filed Oct. 4,2016; the Korean Patent Application Serial No. 10-2016-0129389, filedOct. 6, 2016; and the Korean Patent Application Serial No.10-2017-0090619, filed Jul. 17, 2017. All of these applications areincorporated by reference herein in their entireties.

TECHNICAL FIELD

The present invention relates to image data encoding and decodingtechnology, and more particularly, to a method and apparatus forencoding and decoding a 360-degree image for realistic media service.

BACKGROUND

With the spread of the Internet and mobile terminals and the developmentof information and communication technology, the use of multimedia datais increasing rapidly. Recently, demand for high-resolution images andhigh-quality images such as a high definition (HD) image and an ultrahigh definition (UHD) image is emerging in various fields, and demandfor realistic media service such as virtual reality, augmented reality,and the like is increasing rapidly. In particular, since multi-viewimages captured with a plurality of cameras are processed for 360-degreeimages for virtual reality and augmented reality, the amount of datagenerated for the processing increases massively, but the performance ofan image processing system for processing a large amount of data isinsufficient.

As described above, in an image encoding and decoding method andapparatus of the related art, there is a demand for improvement ofperformance in image processing, particularly, image encoding/decoding.

SUMMARY

It is an object of the present invention to provide a method forimproving an image setting process in initial steps for encoding anddecoding. More particularly, the present invention is directed toproviding an encoding and decoding method and apparatus for improving animage setting process in consideration of the characteristics of a360-degree image.

According to an aspect of the present invention, there is provided amethod of decoding a 360-degree image.

Here, the method of decoding a 360-degree image may include receiving abitstream including an encoded 360-degree image, generating a predictedimage with reference to syntax information acquired from the receivedbitstream, acquiring a decoded image by combining the generatedpredicted image with a residual image acquired by inversely quantizingand inversely transforming the bitstream, and reconstructing the decodedimage into the 360-degree image according to a projection format.

Here, the syntax information may include projection format informationfor the 360-degree image.

Here, the projection format information may be information indicating atleast one of an Equi-Rectangular Projection (ERP) format in which the360-degree image is projected into a 2D plane, a CubeMap Projection(CMP) format in which the 360-degree image is projected to a cube, anOctaHedron Projection (OHP) format in which the 360-degree image isprojected to an octahedron, and an IcoSahedral Projection (ISP) formatin which the 360-degree image is projected to a polyhedron.

Here, the reconstructing may include acquiring arrangement informationaccording to region-wise packing with reference to the syntaxinformation and rearranging blocks of the decoded image according to thearrangement information.

Here, the generating of the predicted image may include performing imageexpansion on a reference picture acquired by restoring the bitstream,and generating a predicted image with reference to the reference pictureon which the image expansion is performed.

Here, the performing of the image expansion may include performing imageexpansion on the basis of partitioning units of the reference picture.

Here, the performing of the image expansion on the basis of thepartitioning units may include generating an expanded regionindividually for each partitioning unit by using the reference pixel ofthe partitioning unit.

Here, the expanded region may be generated using a boundary pixel of apartitioning unit spatially adjacent to a partitioning unit to beexpanded or using a boundary pixel of a partitioning unit having imagecontinuity with a partitioning unit to be expanded.

Here, the performing of the image expansion on the basis of thepartitioning units may include generating an expanded image for a regionwhere two or more partitioning units that are spatially adjacent to eachother among the partitioning units are combined, using a boundary pixelof the combined region.

Here, the performing of the image expansion on the basis of thepartitioning units may include generating an expanded region betweenpartitioning units that are spatially adjacent to each other among thepartitioning units, using all adjacent pixel information of the adjacentpartitioning units.

Here, the performing of the image expansion on the basis of thepartitioning units may include generating the expanded region using anaverage value of adjacent pixels of the spatially adjacent partitioningunits.

Here, the generating of the predicted image may include generating aprediction block of a current block to be decoded using a referenceregion that is included within at least one reference picture and is setin a region having image continuity between faces according to theprojection format, by referring to motion information obtained from thesyntax information.

Here, the reference picture may be expanded in up, down, left, and rightdirections on the basis of image continuity according to the projectionformat, and then the reference region is set.

Here, the reference picture may be expanded in a face unit, and thereference region is set over the face boundary.

Here, the motion information may include at least one among a referencepicture list to which the reference picture belongs, an index of thereference picture, and a motion vector indicating the reference region.

Here, the generating of the prediction block of the current block mayinclude partitioning the current block into a plurality of sub-blocksand generating a prediction block for each of the plurality ofsub-blocks that result from the partitioning.

With the image encoding/decoding method and apparatus according to anembodiment of the present invention, it is possible to enhancecompression performance. In particular, for a 360-degree image, it ispossible to enhance compression performance.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an image encoding apparatus according to anembodiment of the present invention.

FIG. 2 is a block diagram of an image decoding apparatus according to anembodiment of the present invention.

FIG. 3 is an example diagram in which image information is partitionedinto layers in order to compress an image.

FIG. 4 is a conceptual diagram showing examples of image partitioningaccording to an embodiment of the present invention.

FIG. 5 is another example diagram of an image partitioning methodaccording to an embodiment of the present invention.

FIG. 6 is an example diagram of a general image resizing method.

FIG. 7 is an example diagram of image resizing according to anembodiment of the present invention.

FIG. 8 is an example diagram of a method of constructing a regiongenerated through expansion in the image resizing method according to anembodiment of the present invention.

FIG. 9 is an example diagram of a method of constructing a region to bedeleted and a region to be generated in the image resizing methodaccording to an embodiment of the present invention.

FIG. 10 is an example diagram of image reconstruction according to anembodiment of the present invention.

FIG. 11 is an example diagram showing images before and after an imagesetting process according to an embodiment of the present invention.

FIG. 12 is an example diagram of resizing each partitioning unit of animage according to an embodiment of the present invention.

FIG. 13 is an example diagram of a set of resizing or setting of apartitioning unit in an image.

FIG. 14 is an example diagram in which both of a process of resizing animage and a process of resizing partitioning units in an image arerepresented.

FIG. 15 is an example diagram showing a two-dimensional (2D) planarspace and a three-dimensional (3D) space showing a 3D image.

FIGS. 16A to 16D are conceptual diagrams illustrating a projectionformat according to an embodiment of the present invention.

FIG. 17 is a conceptual diagram showing that a projection format isincluded in a rectangular image according to an embodiment of thepresent invention.

FIG. 18 is a conceptual diagram of a method of converting a projectionformat into a rectangular shape, that is, a method of performingrearrangement on a face to exclude a meaningless region according to anembodiment of the present invention.

FIG. 19 is a conceptual diagram showing that a region-wise packingprocess is performed to convert a CMP projection format into arectangular image according to an embodiment of the present invention.

FIG. 20 is a conceptual diagram of 360-degree image partitioningaccording to an embodiment of the present invention.

FIG. 21 is an example diagram of 360-degree image partitioning and imagereconstruction according to an embodiment of the present invention.

FIG. 22 is an example diagram in which an image packed or projected byCMP is partitioned into tiles.

FIG. 23 is a conceptual diagram illustrating an example of resizing a360-degree image according to an embodiment of the present invention.

FIG. 24 is a conceptual diagram illustrating continuity between faces ina projection format (e.g., CHP, OHP, or ISP) according to an embodimentof the present invention.

FIG. 25 is a conceptual diagram illustrating continuity of a face ofSection 21C which is an image acquired through an image reconstructionprocess or a region-wise packing process in the CMP projection format.

FIG. 26 is an example diagram illustrating image resizing in the CMPprojection format according to an embodiment of the present invention.

FIG. 27 is an example diagram illustrating resizing of an imagetransformed and packed in the CMP projection format according to anembodiment of the present invention.

FIG. 28 is an example diagram illustrating a data processing method forresizing a 360-degree image according to an embodiment of the presentinvention.

FIG. 29 is an example diagram showing a tree-based block form.

FIG. 30 is an example diagram showing a type-based block form.

FIG. 31 is an example diagram showing various types of blocks that maybe acquired by a block partitioning part of the present invention.

FIG. 32 is an example diagram illustrating tree-based partitioningaccording to an embodiment of the present invention.

FIG. 33 is an example diagram illustrating tree-based partitioningaccording to an embodiment of the present invention.

FIG. 34 is an example diagram illustrating various cases in which aprediction block is obtained through inter-prediction.

FIG. 35 is an example diagram illustrating composition of a referencepicture list according to an embodiment of the present invention.

FIG. 36 is a conceptual diagram illustrating a non-translation motionmodel according to an embodiment of the present invention.

FIG. 37 is an example diagram illustrating motion estimation insub-block units according to an embodiment of the present invention.

FIG. 38 is an exemplary diagram illustrating that inter-pictureprediction is performed using an expanded picture according to anembodiment of the present invention.

FIG. 39 is a conceptual diagram illustrating an expansion of a face unitaccording to an embodiment of the present invention.

FIG. 40 is an example diagram illustrating that inter-picture predictionis performed using an expanded image according to an embodiment of thepresent invention.

DETAILED DESCRIPTION

Accordingly, while the invention is susceptible to various modificationsand alternative forms, specific embodiments thereof are shown by way ofexample in the drawings and will herein be described in detail. Itshould be understood, however, that there is no intent to limit theinvention to the particular forms disclosed, but on the contrary, theinvention is to cover all modifications, equivalents, and alternativesfalling within the spirit and scope of the invention. Like numbers referto like elements throughout the description of the figures.

It will be understood that, although the terms first, second, etc. maybe used herein to describe various elements, these elements should notbe limited by these terms. These terms are only used to distinguish oneelement from another. For example, a first element could be termed asecond element, and, similarly, a second element could be termed a firstelement, without departing from the scope of the present invention. Asused herein, the term “and/or” includes any and all combinations of oneor more of the associated listed items.

It will be understood that when an element is referred to as being“connected” or “coupled” to another element, it can be directlyconnected or coupled to the other element or intervening elements may bepresent. In contrast, when an element is referred to as being “directlyconnected” or “directly coupled” to another element, there are nointervening elements present. Other words used to describe therelationship between elements should be interpreted in a like fashion(i.e., “between” versus “directly between”, “adjacent” versus “directlyadjacent”, etc.).

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”,“comprising,”, “includes” and/or “including”, when used herein, specifythe presence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

Unless otherwise defined, all terms (including technical and scientificterms) used herein have the same meaning as commonly understood by oneof ordinary skill in the art to which this invention belongs. It will befurther understood that terms, such as those defined in commonly useddictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of the relevant art andwill not be interpreted in an idealized or overly formal sense unlessexpressly so defined herein.

An image encoding apparatus and an image decoding apparatus may each bea user terminal such as a personal computer (PC), a laptop computer, apersonal digital assistant (PDA), a portable multimedia player (PMP), aPlayStation Portable (PSP), a wireless communication terminal, a smartphone, and a TV, a virtual reality (VR) device, an augmented reality(AR) device, a mixed reality (MR) device, a head mounted display (HMD)device, and smart glasses or a server terminal such as an applicationserver and a service server, and may include various devices having acommunication device, such as a communication modem, for communicatingwith various equipment or wired/wireless communication networks, amemory for storing various programs and data used to encode or decode animage or perform inter- or intra-prediction for the encoding ordecoding, a processor for executing programs to perform computation andcontrol operations, and so on. In addition, an image encoded into abitstream by the image encoding apparatus may be transmitted, in realtime or in non-real time, to the image decoding apparatus through awired/wireless communication network such as the Internet, a short-rangewireless network, a wireless local area network (LAN), a WiBro network,a mobile communication network or through a variety of communicationinterfaces such as a cable, a universal serial bus (USB), or the like.Then, the bitstream may be decoded by the image decoding apparatus to berestored and replayed as the image.

Also, the image encoded into the bitstream by the image encodingapparatus may be transferred from the image encoding apparatus to theimage decoding apparatus through a computer-readable recording medium.

The above-described image encoding apparatus and decoding apparatus maybe separate apparatuses, but may be provided as one imageencoding/decoding apparatus according to the implementation. In thiscase, some elements of the image encoding apparatus may be substantiallythe same as those of the image decoding apparatus and may be implementedto include at least the same structures or perform the same functions.

Therefore, in the following detailed description of technical elementsand their working principles, redundant description of the correspondingtechnical elements will be omitted.

Also, the image decoding apparatus corresponds to a computing apparatusthat applies an image encoding method performed by the image encodingapparatus to a decoding process, and thus the following description willfocus on the image encoding apparatus.

The computing apparatus may include a memory configured to store aprogram or a software mode for implementing an image encoding methodand/or an image decoding method and a processor connected to the memoryto execute the program. Also, the image encoding apparatus may also bereferred to as an encoder, and the image decoding apparatus may also bereferred to as a decoder.

Generally, an image may be composed of a series of still images. Thestill images may be classified in units of groups of pictures (GOPs),and each still image may be referred to as a picture. In this case, thepicture may indicate one of a frame and a field in a progressive signaland an interlace signal. The picture may be represented as “frame” whenencoding/decoding is performed on a frame basis and may be representedas “field” when encoding/decoding is performed on a field basis. Thepresent invention assumes a progressive signal, but may also be appliedto an interlace signal. As a higher concept, units such as a GOP and asequence may exist, and also each picture may be partitioned intopredetermined areas such as slices, tiles, blocks, and the like. Also,one GOP may include units such as I-picture, P-picture, and B-picture.I-picture may refer to a picture that is autonomously encoded/decodedwithout using a reference picture, and P-picture and B-picture may referto a picture that is encoded/decoded by performing a process such asmotion estimation and motion compensation using a reference picture.Generally, P-picture may use I-picture and B-picture as referencepictures, and B-picture may use I-picture and P-picture as referencepictures. However, the above definitions may also be changed by settingsof encoding/decoding.

Here, a picture referred to in encoding/decoding is called a referencepicture, and a block or pixel referred to in encoding/decoding is calleda reference block or a reference pixel. Also, reference data may includefrequency-domain coefficients and various types of encoding/decodinginformation generated and determined during an encoding/decodingprocess, as well as spatial-domain pixel values. For example, thereference data may correspond to intra-prediction information or motioninformation in a prediction part, transformation information in atransformation part/an inverse transformation part, quantizationinformation in a quantization part/an inverse quantization part,encoding/decoding information (context information) in an encodingpart/a decoding part, filter information in an in-loop filter part, andthe like.

The minimum unit of the image may be a pixel, and the number of bitsused to represent one pixel is called a bit depth. Generally, the bitdepth may be eight bits, and a bit depth of eight or more bits may besupported depending on the encoding settings. At least one bit depth maybe supported depending on a color space. Also, at least one color spacemay be included according to an image color format. One or more pictureshaving the same size or one or more pictures having different sizes maybe included according to a color format. For example, YCbCr 4:2:0 may becomposed of one luminance component (Y in this example) and twochrominance components (Cb/Cr in this example). At this time, thecomposition ratio of the chrominance components and the luminancecomponent may be 1:2 in width and height. As another example, YCbCr4:4:4 may have the same composition ratio in width and height. Like theabove example, when one or more color spaces are included, a picture maybe partitioned into the color spaces.

The present invention will be described on the basis of any color space(Y in this example) of any color format (YCbCr in this example), andthis description will be applied to another color space (Cb and Cr inthis example) of the color format in the same or similar manner(settings dependent on a specific color space). However, a partialdifference (settings independent of a specific color space) may be givento each color space. That is, the settings dependent on each color spacemay refer to settings proportional to or dependent on the compositionratio of each component (e.g., 4:2:0, 4:2:2, or 4:4:4), and the settingindependent of each color space may refer to settings of only acorresponding color space, independently from or regardless of thecomposition ratio of each component. In the present invention, someelements may have independent settings or dependent settings dependingon the encoder/decoder.

Setting information or syntax elements needed during an image encodingprocess may be determined at a level of units such as a video, asequence, a picture, a slice, a tile, a block, and the like. The unitsinclude a video parameter set (VPS), a sequence parameter set (SPS), apicture parameter set (PPS), a slice header, a tile header, and a blockheader. An encoder may add the units to a bitstream and send thebitstream to a decoder. The decoder may parse the bitstream at the samelevel, restore the setting information sent by the encoder, and use thesetting information in an image decoding process. Also, relatedinformation may be transmitted through a bitstream in the form ofsupplement enhancement information (SEI) or metadata, and then may beparsed and then used. Each parameter set has a unique ID value, and alower parameter set may have an ID value of an upper parameter set to bereferred to. For example, a lower parameter set may refer to informationof an upper parameter set having a corresponding ID value among one ormore upper parameter sets. Among various examples of the above-describedunits, when any one unit includes one or more different units, the anyone unit may be referred to as an upper unit, and the included units maybe referred to as a lower unit.

Setting information having occurred in such a unit may include settingsindependent of each unit or settings dependent on a previous, following,or upper unit. Here, it will be understood that the dependent settingsindicate setting information of a corresponding unit using flaginformation corresponding to settings of the previous, following, orupper unit (e.g., 1-bit flag; 1 indicates Follow, and 0 indicates Do NotFollow). In the present invention, the setting information will bedescribed, focusing on an example of the independent settings. However,an example may also be included in which a relation dependent on thesetting information of the previous, following, or upper unit of thecurrent unit is added to, or substituted for, the independent settings.

FIG. 1 is a block diagram of an image encoding apparatus according to anembodiment of the present invention. FIG. 2 is a block diagram of animage decoding apparatus according to an embodiment of the presentinvention.

Referring to FIG. 1 , the image encoding apparatus may be configured toinclude a prediction part, a subtractor, a transformation part, aquantization part, an inverse quantization part, an inversetransformation part, an adder, an in-loop filter part, a memory, and/oran encoding part, some of which may not necessarily be included. Some orall of the elements may be included selectively depending on theimplementation, and some additional elements which are not shown hereinmay be included.

Referring to FIG. 2 , the image decoding apparatus may be configured toinclude a decoding part, a prediction part, an inverse quantizationpart, an inverse transformation part, an adder, an in-loop filter part,and/or a memory, some of which may not necessarily be included. Some orall of the elements may be included selectively depending on theimplementation, and some additional elements which are not shown hereinmay be included.

The image encoding apparatus and decoding apparatus may be separateapparatuses, but may be provided as one image encoding/decodingapparatus depending on the implementation. In this case, some elementsof the image encoding apparatus may be substantially the same as thoseof the image decoding apparatus and may be implemented to include atleast the same structures or perform the same functions. Therefore, inthe following detailed description of technical elements and theirworking principles, redundant description of the corresponding technicalelements will be omitted. The image decoding apparatus corresponds to acomputing apparatus that applies an image encoding method performed bythe image encoding apparatus to a decoding process, and thus thefollowing description will focus on the image encoding apparatus. Theimage encoding apparatus may also be referred to as an encoder, and theimage decoding apparatus may also be referred to as a decoder.

The prediction part may be implemented using a prediction module and maygenerate a prediction block by performing intra-prediction orinter-prediction on a block to be encoded. The prediction part generatesthe prediction block by predicting a current block to be encoded in animage. In other words, the prediction part may predict pixel values ofpixels of a current block to be encoded in an image throughintra-prediction or inter-prediction to generate a prediction blockhaving predicted pixel values of the pixels. Also, the prediction partmay deliver information needed to generate the prediction block to theencoding part so that prediction mode information is encoded. Theencoding part adds corresponding information to a bitstream andtransmits the bitstream to the decoder. The decoding part of the decodermay parse the corresponding information, restore the prediction modeinformation, and then use the prediction mode information to performintra-prediction or inter-prediction.

The subtractor subtracts the prediction block from the current block togenerate a residual block. In other words, the subtractor may calculatea difference between a pixel value of each pixel of the current block tobe encoded and a predicted pixel value of each pixel of the predictionblock generated through the prediction part to generate a residualblock, which is a block-type residual signal.

The transformation part may transform a signal belonging to a spacedomain into a signal belonging to a frequency domain. In this case, asignal acquired through the transformation process is called atransformed coefficient. For example, the residual block with theresidual signal delivered from the subtractor may be transformed into atransformation block with a transformed coefficient. In this case, aninput signal is determined according to encoding settings and is notlimited to the residual signal.

The transformation part may perform transformation on the residual blockby using a transformation technique such as Hadamard Transform, DiscreteSine Transform (DST)-based transformation, and Discrete Cosine Transform(DCT)-based transformation. However, the present invention is notlimited thereto, and various enhanced and modified transformationtechniques may be used.

For example, at least one of the transformation techniques may besupported, and at least one detailed transformation technique may besupported in each transformation technique. In this case, the at leastone detailed transformation technique may be a transformation techniquein which some base vectors are differently constructed in eachtransformation technique. For example, as the transformation techniques,DST-based transformation and DCT-based transformation may be supported.Detailed transformation techniques such as DST-I, DST-II, DST-III,DST-V, DST-VI, DST-VII, and DST-VIII may be supported for DST, anddetailed transformation techniques such as DCT-I, DCT-II, DCT-III,DCT-V, DCT-VI, DCT-VII, and DCT-VIII may be supported for DCT.

One of the transformation techniques may be set as a defaulttransformation technique (e.g., one transformation technique && onedetailed transformation technique), and additional transformationtechniques may be supported (e.g., a plurality of transformationtechniques∥a plurality of detailed transformation techniques). Whetherto support an additional transformation technique may be determined inunits of sequences, pictures, slices, or tiles, and related informationmay be generated according to the units. When an additionaltransformation technique is supported, transformation techniqueselection information may be determined in block units, and relatedinformation may be generated.

The transformation may be performed horizontally and/or vertically. Forexample, two-dimensional (2D) transformation is performed byhorizontally and vertically performing one-dimensional (1D)transformation using a base vector so that a pixel value in a spatialdomain may be transformed into a frequency domain.

Also, the transformation may be performed horizontally and/or verticallyin an adaptive manner. In detail, whether to perform the transformationin the adaptive manner may be determined according to at least oneencoding setting. For the intra-prediction, for example, DCT-I may beapplied horizontally and DST-I may be applied vertically when theprediction mode is a horizontal mode, DST-VI may be applied horizontallyand DOT-VI may be applied vertically when the prediction model is avertical mode, DCT-II may be applied horizontally and DCT-V may beapplied vertically when the prediction mode is Diagonal down left, andDST-I may be applied horizontally and DST-VI may be applied verticallywhen the prediction mode is Diagonal down right.

The sizes and forms of transformation blocks may be determined accordingto encoding costs for candidates of the size and shape of thetransformation blocks. Image data of the transformation blocks andinformation regarding the determined sizes and forms of thetransformation blocks may be encoded.

Among the transformation forms, square transformation may be set as adefault transformation form, and an additional transformation form(e.g., a rectangular form) may be supported. Whether to support anadditional transformation form may be determined in units of sequences,pictures, slices, or tiles, and related information may be generatedaccording to the units. Transformation form selection information may bedetermined in block units, and related information may be generated.

Also, whether to support a transformation block form may be determinedaccording to encoding information. In this case, the encodinginformation may correspond to a slice type, an encoding mode, the sizeand shape of a block, a block partitioning scheme, etc. That is, onetransformation form may be supported according to at least one piece ofencoding information, and a plurality of transformation forms may besupported according to at least one piece of encoding information. Theformer case may be an implicit situation, and the latter case may be anexplicit situation. For the explicit situation, adaptive selectioninformation indicating an optimal candidate group selected from among aplurality of candidate groups may be generated and added to a bitstream.According to the present invention, in addition to this example, it willbe understood that when encoding information is explicitly generated,the information is added to a bitstream in various units and relatedinformation is parsed in various units and restored into decodinginformation by the decoder. Also, it will be understood that whenencoding/decoding information is implicitly processed, the processing isperformed through the same process, rule, and the like by the encoderand decoder.

As an example, the support of the rectangular transformation may bedetermined according to a slice type. A transformation form supportedfor I-slice may be square transformation, and a transformation formsupported for P/B-slice may be square or rectangular transformation.

As an example, the support of the rectangular transformation may bedetermined according to an encoding mode. A transformation formsupported for intra-prediction may be square transformation, and atransformation form supported for inter-prediction may be squaretransformation and/or rectangular transformation.

As an example, the support of the rectangular transformation may bedetermined according to the size and shape of a block. A transformationform supported by a block of a certain size or greater may be squaretransformation, and a transformation form supported by a block of lessthan a certain size may be square transformation and/or rectangulartransformation.

As an example, the support of the rectangular transformation may bedetermined according to a block partitioning scheme. When a block to betransformed is a block acquired through a quad-tree partitioning scheme,the supported transformation form may be square transformation. When ablock to be transformed is a block acquired through a binary treepartitioning scheme, the supported transformation form may be squaretransformation or rectangular transformation.

The above example may be an example of the support of the transformationform according to one piece of encoding information, and a plurality ofpieces of information may be associated with additional transformationform support settings in combination. The above example is merely anexample of the additional transformation form support according tovarious encoding settings. However, the present invention is not limitedthereto, and various modifications may be made thereto.

The transformation process may be omitted according to encoding settingsor image characteristics. For example, the transformation process(including the inverse process) may be omitted according to encodingsettings (e.g., in this example, a lossless compression environment isassumed). As another example, the transformation process may be omittedwhen compression performance through transformation is not shownaccording to the image characteristics. In this case, the transformationmay be omitted for all the units or one of the horizontal unit and thevertical unit. Whether to support the omission may be determinedaccording to the size and shape of a block.

For example, it is assumed that horizontal transformation and verticaltransformation are set to be jointly omitted. The transformation may beperformed neither horizontally nor vertically when a transformationomission flag is 1, and the transformation may be performed bothhorizontally and vertically when the transformation omission flag is 0.On the other hand, it is assumed that horizontal transformation andvertical transformation are set to be independently omitted. Thehorizontal transformation is not performed when a first transformationomission flag is 1, and the horizontal transformation is performed whenthe first transformation omission flag is 0. Then verticaltransformation is not performed when a second transformation omissionflag is 1, and the vertical transformation is performed when the secondtransformation omission flag is 0.

The omission of the transformation may be supported when the size of ablock corresponds to a range A, and the omission of the transformationcannot be supported when the size of a block corresponds to a range B.For example, when the width of a block is greater than M or the heightof a block is greater than N, the transformation omission flag cannot besupported. When the width of a block is less than m or the height of ablock is less than n, the transformation omission flag may be supported.M(m) and N(n) may be the same as or different from each other. Settingsassociated with the transformation may be determined in units ofsequences, pictures, slices, or the like.

When an additional transformation technique is supported, atransformation technique setting may be determined according to at leastone piece of encoding information. In this case, the encodinginformation may correspond to a slice type, an encoding mode, the sizeand shape of a block, a prediction mode, etc.

As an example, the support of the transformation technique may bedetermined according to the encoding mode. The transformation techniquesupported for the intra-prediction may include DCT-I, DCT-III, DCT-VI,DST-II, and DST-III, and the transformation technique supported for theinter-prediction may include DCT-II, DCT-III, and DST-III.

As an example, the support of the transformation technique may bedetermined according to the slice type. The transformation techniquesupported for I-slice may include DCT-I, DCT-II, and DCT-III, thetransformation technique supported for P-slice may include DCT-V, DST-V,and DST-VI, and the transformation technique supported for B-slice mayinclude DCT-I, DCT-II, and DST-III.

As an example, the support of the transformation technique may bedetermined according to the prediction mode. The transformationtechnique supported by a prediction mode A may include DCT-I and DCT-II,the transformation technique supported by a prediction mode B mayinclude DCT-I and DST-I, and the transformation technique supported by aprediction mode C may include DCT-I. In this case, the prediction mode Aand the prediction mode B may be each a directional mode, and thepredication mode C may be a non-directional mode.

As an example, the support of the transformation technique may bedetermined according to the size and shape of a block. Thetransformation technique supported by a block of a certain size orgreater may include DCT-II, the transformation technique supported by ablock of less than a certain size may include DCT-II and DST-V, and thetransformation technique supported by a block of a certain size orgreater and less than a certain size may include DCT-I, DCT-II, andDST-I. Also, the transformation technique supported in a square shapeform may include DCT-I and DCT-II, and the transformation techniquesupported in a rectangular shape may include DCT-I and DST-I.

The above example may be an example of the support of the transformationtechnique according to one piece of encoding information, and aplurality of pieces of information may be associated with additionaltransformation technique support settings in combination. The presentinvention is not limited to the above example, and modifications may bemade thereto. Also, the transformation part may deliver informationneeded to generate a transformation block to the encoding part so thatthe information is encoded. The encoding part adds correspondinginformation to a bitstream and transmits the bitstream to the decoder.The decoding part of the decoder may parse the information and use theparsed information in the inverse transformation process.

The quantization part may quantize input signals. In this case, a signalacquired through the quantization process is called a quantizedcoefficient. For example, the quantization part may quantize a residualblock with a residual transformation coefficient delivered from thetransformation part and thus acquire a quantization block with aquantization coefficient. In this case, the input signal is determinedaccording to encoding settings and is not limited to the residualtransformation coefficient.

The quantization part may use a quantization technique such as Dead ZoneUniform Threshold Quantization, Quantization Weighted Matrix, or thelike to quantize the transformed residual block. However, the presentinvention is not limited thereto, and various quantization techniquesthat are improved and modified may be used. Whether to support anadditional quantization technique may be determined in units ofsequences, pictures, slices, or tiles, and related information may begenerated according to the units. When an additional quantizationtechnique is supported, quantization technique selection information maybe determined in block units, and related information may be generated.

When an additional quantization technique is supported, a quantizationtechnique setting may be determined according to at least one piece ofencoding information. In this case, the encoding information maycorrespond to a slice type, an encoding mode, the size and shape of ablock, a prediction mode, etc.

For example, the quantization part may differently set a quantizationweighted matrix corresponding to an encoding mode and a weighted matrixapplied according to the inter-prediction/intra-prediction. Also, thequantization part may differently set a weighted matrix appliedaccording to an intra-prediction mode. In this case, when it is assumedthat the quantization weighted matrix has a size of M×N, which is thesame as the size of the quantization block, the quantization weightedmatrix may be a quantization matrix in which some quantizationcomponents are differently constructed.

The quantization process may be omitted according to encoding settingsor image characteristics. For example, the quantization process(including the inverse process) may be omitted according to encodingsettings (e.g., e.g., in this example, a lossless compressionenvironment is assumed). As another example, the quantization processmay be omitted when compression performance through quantization is notshown according to the image characteristics. In this case, some or allof the regions may be omitted, and whether to support the omission maybe determined according to the size and shape of a block.

Information regarding quantization parameters (QPs) may be generated inunits of sequences, pictures, slices, tiles, or blocks. For example, adefault QP may be set in an upper unit in which the QP information isfirst generated <1>, and a QP may be set to a value that is the same asor different from that of the QP set in the upper unit. In thequantization process performed in some units through the process, the QPmay be finally determined. In this case, the unit such as a sequence anda picture may be an example corresponding to <1>, the unit such as aslice, a tile, and a block may be an example corresponding to <2>, andthe unit such as a block may be an example corresponding to <3>.

The information regarding the QP may be generated on the basis of a QPin each unit. Alternatively, a predetermined QP may be set as apredicted value, and information regarding differences from the QPs inthe units may be generated. Alternatively, a QP acquired based on atleast one of a QP set in an upper unit, a QP set in the same andprevious unit, or a QP set in a neighboring unit may be set as apredicted value, and information regarding a difference from a QP in thecurrent unit may be generated. Alternatively, a QP set in an upper unitand a QP acquired based on at least one piece of encoding informationmay be set as predicted values, and difference information from the QPin the current unit may be generated. In this case, the same andprevious unit may be a unit that may be defined in an order of encodingthe units, the neighboring unit may be a spatially adjacent unit, andthe encoding information may be a slice type, an encoding mode, aprediction mode, location information, etc. of a corresponding unit.

As an example, the QP in the current unit may be used to set the QP inthe upper unit as a predicted value and generate difference information.Information regarding a difference between a QP set in a slice and a QPset in a picture may be generated, or information regarding a differencebetween a QP set in a tile and a QP set in a picture may be generated.Also, information regarding a difference between a QP set in a block andthe QP set in the slice or tile may be generated. Also, informationregarding a difference between a QP set in a sub-block and the QP set inthe block may be generated.

As an example, the QP in the current unit may be used to set a QPacquired based on a QP in at least one neighboring unit or a QP in atleast one previous unit as a predicted value and generate differenceinformation. Information regarding a difference from a QP acquired basedon a QP of a neighboring block, such as a block on a left side, an upperleft side, a lower left side, an upper side, an upper right side, andthe like of the current block may be generated. Alternatively,information regarding a difference from a QP of an encoded picturebefore the current picture may be generated.

As an example, the QP in the current unit may be used to set a QP in anupper unit and a QP acquired based on at least one piece of encodinginformation as predicted values and generate difference information.Also, information regarding a difference between the QP in the currentbock and a QP of a slice corrected according to a slice type (I/P/B) maybe generated. Alternatively, information regarding a difference betweenthe QP in the current bock and a QP of a tile corrected according to theencoding mode (intra/inter) may be generated. Alternatively, informationregarding a difference between the QP in the current bock and a QP of apicture corrected according to the prediction mode(directionality/non-directionality) may be generated. Alternatively,information regarding a difference between the QP in the current bockand a QP of a picture corrected according to location information (x/y)may be generated. In this case, the correction may refer to an operationof adding or subtracting an offset to or from a QP in an upper unit usedfor prediction. In this case, at least one piece of offset informationmay be supported according to encoding settings, and information that isimplicitly processed or explicitly associated may be generated accordingto a predetermined process. The present invention is not limited to theabove example, and modifications may be made thereto.

The above example may be an example that is allowed when a signalindicating QP variation is provided or activated. For example, when thesignal indicating QP variation is neither provided nor activated, thedifference information is not generated, and the predicted QP may bedetermined as a QP in each unit. As another example, when the signalindicating QP variation is provided or activated, the differenceinformation is generated, and the predicted QP may be determined as a QPin each unit when the difference information has a value of 0.

The quantization part may deliver information needed to generate aquantization block to the encoding part so that the information isencoded. The encoding part adds corresponding information to a bitstreamand transmits the bitstream to the decoder. The decoding part of thedecoder may parse the information and use the parsed information in theinverse quantization process.

The above example has been described under the assumption that aresidual block is transformed and quantized through the transformationpart and the quantization part. However, a residual signal of theresidual block may be transformed into a residual block with atransformation efficient while the quantization process is notperformed. Alternatively, only the quantization process may be performedwhile the residual signal of the residual block is not transformed intoa transformation coefficient. Alternatively, neither the transformationprocess nor the quantization process may be performed. This may bedetermined according to encoding settings.

The encoding part may scan a quantization coefficient, a transformationcoefficient, or a residual signal of the generated residual block in atleast one scan order (e.g., zigzag scanning, vertical scanning,horizontal scanning, etc.), generate a quantization coefficient string,a transformation coefficient string, or a signal string, and encode thequantization coefficient string, transformation coefficient string, orsignal string using at last one entropy coding technique. In this case,information regarding the scan order may be determined according toencoding settings (e.g., an encoding mode, a prediction mode, etc.) andmay be used to generate information that is implicitly determined orexplicitly associated. For example, one scanning order may be selectedfrom among a plurality of scanning orders according to theintra-prediction mode.

Also, the encoding part may generate encoding data including encodinginformation delivered from each element and may output the encoding datain a bitstream. This may be implemented with a multiplexer (MUX). Inthis case, the encoding may be performed using a method such asExponential Golomb, Context Adaptive Variable Length Coding (CAVLC), andContext Adaptive Binary Arithmetic Coding (CABAC) as an encodingtechnique. However, the present invention is not limited thereto, andvarious encoding techniques obtained by improving and modifying theabove encoding techniques may be used.

When entropy encoding (e.g., CABAC in this example) is performed on asyntax element such as information generated through anencoding/decoding process and the residual block data, an entropyencoding apparatus may include a binarizer, a context modeler, and abinary arithmetic coder. In this case, the binary arithmetic coder mayinclude a regular coding engine and a bypass coding engine.

A syntax element input to the entropy encoding apparatus may not be abinary value. Thus, when syntax elements are not binary values, thebinarizer may binarize the syntax elements and output a bin stringcomposed of 0s or 1s. In this case, a bin represents a bit composed of 0or 1 and may be encoded through the binary arithmetic coder. In thiscase, one of the regular coding engine and the bypass coding engine maybe selected on the basis of the probability of occurrence of 0 and 1 andthis may be determined according to encoding/decoding settings. When asyntax element is data having the frequency of 0 equal to the frequencyof 1, the bypass coding engine may be used; otherwise, the regularcoding engine may be used.

When the syntax element is binarized, various methods may be used. Forexample, Fixed Length Binarization, Unary Binarization, Truncated RiceBinarization, K-th Exp-Golomb binarization, and the like may be used.Also, signed binarization or unsigned binarization may be performeddepending on the range of the value of the syntax element. Thebinarization process for the syntax elements according to the presentinvention may include an additional binarization method as well as thebinarization described in the above example.

The inverse quantization part and the inverse transformation part may beimplemented by inversely perform the processes performed in thetransformation part and the quantization part. For example, the inversequantization part may inversely quantize a transformation coefficientquantized by the quantization part, and the inverse transformation partmay inversely transform the inversely quantized transformationcoefficient to generate a restored residual block.

The adder adds the prediction block and the restored residual block torestore a current block. The restored block may be stored in the memoryand may be used as reference data (for the prediction part, the filterpart, etc.).

The in-loop filter part may additionally perform a post-processingfiltering process of one or more of a deblocking filter, a sampleadaptive offset (SAO), an adaptive loop filter (ALF), and the like. Thedeblocking filter may remove block distortion generated at a boundarybetween blocks from a restored image. The ALF may perform filtering onthe basis of a value obtained by comparing an input image to a restoredimage. In detail, the ALF may perform filtering on the basis of a valueobtained by comparing an input image to an image restored after a blockis filtered through the deblocking filter. Alternatively, the ALF mayperform filtering on the basis of a value obtained by comparing an inputimage to an image restored after a block is filtered through the SAO.The SAO may restore an offset difference on the basis of a valueobtained by comparing an input image to a restored image and may beapplied in the form of band offset (BO), edge offset (EO), and the like.In detail, the SAO may add an offset against an original image to therestored image, to which the deblocking filter is applied, in units ofat least one pixel and may be applied in the form of BO, EO, and thelike. In detail, the SAO may add an offset against an original image toan image restored after a block is filtered through the ALF in pixelunits and may be applied in the form of BO, EO, and the like.

As filtering information, setting information regarding whether tosupport each post-processing filter may be generated in units ofsequences, pictures, slices, tiles, or the like. Also, the settinginformation regarding whether to execute each post-processing filter maybe generated in units of pictures, slices, tiles, blocks, or the like.The range in which the filter is performed may be classified into theinside of an image and the boundary of an image. The setting informationconsidering the classification may be generated. Also, informationregarding the filtering operation may be generated in units of pictures,slices, tiles, blocks, or the like. The information may be implicitly orexplicitly processed, and an independent filtering process or adependent filtering process may be applied to the filtering depending ona color component, and this may be determined according to encodingsettings. The in-loop filter part may deliver the filtering informationto the encoding part so that the information is encoded. The encodingpart adds corresponding information to a bitstream and transmits thebitstream to the decoder. The decoding part of the decoder may parse theinformation and apply the parsed information to the in-loop filter part.

The memory may store the restored block or picture. The restored blockor picture stored in the memory may be provided to the prediction part,which performs intra-prediction or inter-prediction. In detail, for theprocessing, a space in which a bitstream compressed by an encoder isstored in the form of queues may be set as a coded picture buffer (CPB),and a space in which the decoded image is stored in picture units may beset as a decoded picture buffer (DPB). The CPB may store the decodingparts in the decoding order, emulate the decoding operation in theencoder, and store the compressed bitstream through the emulationprocess. The bitstream output from the CPB is restored through thedecoding process, and the restored image is stored in the DPB, andpictures stored in the DPB may be referred to during an imageencoding/decoding process.

The decoding part may be implemented by inversely performing the processof the encoding part. For example, the decoding part may receive aquantization coefficient string, a transformation coefficient string, ora signal string from the bitstream, decode the string, parse decodingdata including decoding information, and deliver the parsed decodingdata to each element.

Next, an image setting process applied to the image encoding/decodingapparatus according to an embodiment of the present invention will bedescribed. This is an example (initial image settings) applied beforeencoding/decoding, but some processes may be examples to be applied tothe other steps (e.g., steps after the encoding/decoding or sub-steps ofthe encoding/decoding). The image setting process may be performed inconsideration of network and user environments such as multimediacontent characteristics, bandwidths, user terminal performance, andaccessibility. For example, image partitioning, image resizing, imagereconstruction, and the like may be performed according toencoding/decoding settings. The following description of the imagesetting process focuses on a rectangular image. However, the presentinvention is not limited thereto, and the image setting process may beapplied to polygonal images. The same image settings may be appliedirrespective of the image form or different image settings may beapplied, which may be determined according to encoding/decodingsettings. For example, after information regarding the image shape(e.g., a rectangular shape or a non-rectangular shape) is checked,information regarding corresponding image settings may be constructed.

The following example will be described under the assumption thatdependent settings are provided to a color space. However, independentsettings may be provided to the color space. Also, in the followingexample, the independent settings may include independently providingencoding/decoding settings to each color space. Although one color spaceis described, it is assumed that an example in which the description isapplied to another color space (e.g., an example in which N is generatedin the chrominance component when M is generated in the luminancecomponent) is included, and this may be derived. Also, the dependentsettings may include an example in which settings are made in proportionto a color format composition ratio (e.g., 4:4:4, 4:2:2, 4:2:0, etc.)(for example, for 4:2:0, M/2 in the chrominance component in the case ofM in the luminance component). It is assumed that an example in whichthe description is applied to each color space is included, and this maybe derived. This description is not limited to the above example and maybe applied in common to the present invention.

Some constructions in the following example may be applied to variousencoding techniques such as spatial domain encoding, frequency domainencoding, block-based encoding, object-based encoding, and the like.

Generally, an input image may be encoded or decoded as it is or afterimage partitioning. For example, the partitioning may be performed forerror robustness or the like in order to prevent damage caused by packetloss during transmission. Alternatively, the partitioning may beperformed in order to classify regions having different properties inthe same image according to the characteristics, type, and the like ofthe image.

According to the present invention, the image partitioning process mayinclude a partitioning process and an inverse partitioning process. Thefollowing example description will focus on the partitioning process,but the inverse partitioning process may be inversely derived from thepartitioning process.

FIG. 3 is an example diagram in which image information is partitionedinto layers in order to compress an image.

Section 3A is an example diagram in which an image sequence is composedof a plurality of GOP. Also, one GOP may be composed of I-pictures,P-pictures, and B-pictures, as shown in Section 3B. One picture may becomposed of slices, tiles, and the like, as shown in Section 3C. Aslice, tile, or the like may be composed of a plurality of defaultencoding parts, as shown in Section 3D, and a default encoding part maybe composed of at least one encoding sub-unit, as shown in section 3E.The image setting process according to the present invention will bedescribed on the basis of an example to be applied to a unit such as apicture, a slice, and a tile, as shown in Sections 3B and 3C.

FIG. 4 is a conceptual diagram showing examples of image partitioningaccording to an embodiment of the present invention.

Section 4A is a conceptual diagram in which an image (e.g., a picture)is laterally and longitudinally partitioned at regular intervals. Apartitioned region may be referred to as a block. Each block may be adefault encoding part (or a maximum encoding part) acquired through apicture partitioning part and may be a basic unit to be applied to apartitioning unit, which will be described below.

Section 4B is a conceptual diagram in which an image is partitioned inat least one direction selected from a lateral direction and alongitudinal direction. Partitioned regions T₀ to T₃ may be referred toas tiles, and each region may be encoded or decoded independently ordependently from the other regions.

Section 4C is a conceptual diagram in which an image is partitioned intogroups of consecutive blocks. Partitioned regions S₀ and S₁ may bereferred to as slices, and each region may be encoded or decodedindependently or dependently from the other regions. A group ofconsecutive blocks may be defined according to a scan order. Generally,a group of consecutive blocks conforms to raster scan order. However,the present invention is not limited thereto, and a group of consecutiveblocks may be determined according to encoding/decoding settings.

Section 4D is a conceptual diagram in which an image is partitioned intogroups of blocks according to any user-defined settings. Partitionedregions A₀ to A₂ may be referred to as arbitrary partitions, and eachregion may be encoded or decoded independently or dependently from theother regions.

The independent encoding/decoding may denote that when some units (orregions) are encoded or decoded, data in other units cannot bereferenced. In detail, pieces of information used or generated duringtexture encoding and entropy encoding for some units may beindependently encoded without being referenced to one another. Even inthe decoder, for texture decoding and entropy decoding for some units,parsing information and restoration information in other units may notbe referenced to each other. In this case, whether to reference data inother units (or regions) may be limited in a spatial region (e.g.,between regions in one image), but may also be limited in a temporalregion (e.g., between consecutive images or between frames) according toencoding/decoding settings. For example, when some units of the currentimage and some units of another image have continuity or have the sameencoding environments, a reference may be made; otherwise, the referencemay be limited.

Also, the dependent encoding/decoding may denote that when some unitsare encoded or decoded, data in other units can be referenced. Indetail, pieces of information used or generated during texture encodingand entropy encoding for some units may be dependently encoded alongwith being referenced to one another. Even in the decoder, for texturedecoding and entropy decoding for some units, parsing information andrestoration information in other units may be referenced to each other.That is, the above settings may be the same as or similar to those ofgeneral encoding/decoding. In this case, in order to identify a region(here, a face <Face> and the like generated according to a projectionformat), the region may be partitioned according to the characteristics,type, and the like of the image (e.g., a 360-degree image).

In the above example, independent encoding/decoding settings (e.g.,independent slice segments) may be provided to some units (a slice, atile, and the like), and dependent encoding/decoding settings (e.g.,dependent slice segments) may be provided to other units. According tothe present invention, the following description will focus on theindependent encoding/decoding settings.

As shown in Section 4A, a default encoding part acquired through thepicture partitioning part may divided into default encoding blocksaccording to a color space, and may have a size and shape determinedaccording to the characteristics and resolution of the image. Thesupported size or shape of the block may be an N×N square (2^(n)×2^(n);256×256, 128×128, 64×64, 32×32, 16×16, 8×8, etc.; n is an integerranging from 3 to 8) having a width and a height represented as theexponent of 2 (2^(n)) or an M×N rectangle (2^(m)×2^(n)). For example, aninput image may be partitioned into 128×128 for an 8k UHD image, 64×64for a 1080p HD image, or 16×16 for a WVGA image depending on theresolution and may be partitioned into 256×256 for a 360-degree imagedepending on the image type. The default encoding part may bepartitioned into encoding sub-units and then encoded or decoded.Information regarding the default encoding part may be added to abitstream in units of sequences, pictures, slices, tiles, or the like,and may be parsed by the decoder to restore related information.

The image encoding method and the image decoding method according to anembodiment of the present invention may include the following imagepartitioning steps. In this case, the image partitioning process mayinclude an image partitioning indication step, an image partitioningtype identification step, and an image partitioning performing step.Also, the image encoding apparatus and the image decoding apparatus maybe configured to include an image partitioning indication part, an imagepartitioning type identification part, and an image partitioningperforming part, which perform the image partitioning indication step,the image partitioning type identification step, and the imagepartitioning performing step, respectively. For the encoding, a relatedsyntax element may be generated. For the decoding, a related syntaxelement may be parsed.

In the block partitioning process, as shown in Section 4A, the imagepartitioning indication part may be omitted. The image partitioning typeidentification part may check information regarding the size and shapeof a block, and the image partitioning part may perform partitioningthrough identified partitioning type information in the default encodingparts.

A block may be a unit to be always partitioned, but whether to partitionthe other partitioning units (a tile, a slice, and the like) may bedetermined according to encoding/decoding settings. As default settings,the picture partitioning part may perform partitioning in block unitsand then perform partitioning in other units. In this case, the blockpartitioning may be performed on the basis of a picture size.

Also, the partitioning may be performed in block units after beingperformed in other units (tiles, slices, or the like). That is, theblock partitioning may be performed on the basis of the size of apartitioning unit. This may be determined through explicit or implicitprocessing according to encoding/decoding settings. The followingexample description assumes the former case and also will focus in unitsother than blocks.

In the image partitioning indication step, whether to perform imagepartitioning may be determined. For example, when a signal indicatingimage partitioning (e.g., tiles_enabled_flag) is confirmed, thepartitioning may be performed. When the signal indicating imagepartitioning is not confirmed, the partitioning may not be performed, orthe partitioning may be performed by confirming other encoding/decodinginformation.

In detail, it is assumed that the signal indicating image partitioning(e.g., tiles_enabled_flag) is confirmed. When the signal is activated(e.g., tiles_enabled_flag=1), the partitioning may be performed in aplurality of units. When the signal is deactivated (e.g.,tiles_enabled_flag=0), the partitioning may not be performed.Alternatively, the signal indicating image partitioning not beingconfirmed may denote that the partitioning is not performed or isperformed in at least one unit. Whether to perform partitioning in aplurality of units may be confirmed through another signal (e.g.,first_slice_segment_in_pic_flag).

In summary, when the signal indicating image partitioning is provided,the corresponding signal is a signal for indicating whether to performthe partitioning in a plurality of units. Whether to partition thecorresponding image may be determined according to the signal. Forexample, it is assumed that tiles_enabled_flag is a signal indicatingwhether to partition an image. Here, tiles_enabled_flag being equal to 1may denote that an image is partitioned into a plurality of tiles, andtiles_enabled_flag being equal to 0 may denote that an image is notpartitioned.

In summary, when the signal indicating image partitioning is notprovided, the partitioning may not be performed, or whether to partitiona corresponding image may be determined by another signal. For example,first_slice_segment_in_pic_flag is not a signal indicating whether toperform image partitioning but a signal indicating the first slicesegment in an image. Thus, whether to perform partitioning in two ormore units (e.g., the flag being 0 denotes that the image is partitionedinto a plurality of slices) may be confirmed.

The present invention is not limited to the above example, andmodifications may be made thereto. For example, a signal indicatingimage partitioning may not be provided for each tile and may be providedfor each slice. Alternatively, the signal indicating image partitioningmay be provided on the basis of the type, characteristics, and the likeof an image.

In the image partitioning type identification step, an imagepartitioning type may be identified. The image partitioning type may bedefined by a partitioning method, partitioning information, and thelike.

In Section 4B, the tile may be defined as a unit acquired by lateral andlongitudinal partitioning. In detail, the tile may be defined as a groupof adjacent blocks in a quadrilateral space partitioned by at least onelateral or longitudinal partitioning line passing through an image.

Tile partitioning information may include boundary location informationfor a column and a row, tile number information for a column and a row,tile size information, and the like. The tile number information mayinclude the number of columns for the tiles (e.g., num_tile_columns) andthe number of rows for the tiles (e.g., num_tile_rows). Thus, the imagemay be partitioned into a number (=the number of columns×the number ofrows) of tiles. The tile size information may be acquired on the basisof the tile number information. The width or height of the tile may beuniform or non-uniform, and thus under predetermined rules, relatedinformation (e.g., uniform_spacing_flag) may be implicitly determined orexplicitly generated. Also, the tile size information may include sizeinformation of each column and each row of a tile (e.g.,column_width_tile[i] and row_height_tile[i]) or include size informationof the width and height of each tile. Also, the size information may beinformation that may be additionally generated according to whether atile size is uniform (e.g., when the partitioning is non-uniform becauseuniform_spacing_flag is 0).

In Section 4C, a slice may be defined as a unit of grouping consecutiveblocks. In detail, the slice may be defined as a group of consecutiveblocks in predetermined scan order (here, in raster scan).

Slice partitioning information may include slice number information,slice location information (e.g., slice_segment_address), and the like.In this case, the slice location information may be location informationof a predetermined block (e.g., the first rank in scan order in theslice). In this case, the location information may be block scan orderinformation.

In Section 4D, various partitioning settings are allowed for thearbitrary partition.

In Section 4D, a partitioning unit may be defined as a group of blocksthat are spatially adjacent to one another, and information regardingthe partitioning may include information regarding the size, form, andlocation of the partitioning unit. This is merely an example of thearbitrary partition, and various partitioning forms may be allowed asshown in FIG. 5 .

FIG. 5 is another example diagram of an image partitioning methodaccording to an embodiment of the present invention.

In Sections 5A and 5B, an image may be laterally or longitudinallypartitioned into a plurality of regions at at least one block interval,and the partitioning may be performed on the basis of block locationinformation. Section 5A illustrates examples A₀ and A₁ in which thepartitioning is laterally performed on the basis of row information ofeach block, and Section 5B illustrates examples B₀ to B₃ in which thepartitioning is laterally and longitudinally performed on the basis ofcolumn information and row information of each block. Informationregarding the partitioning may include the number of partitioning units,block interval information, a partitioning direction, etc., and when thepartitioning information is implicitly included according to apredetermined rule, some of the partitioning information may not begenerated.

In Sections 5C and 5D, an image may be partitioned into groups ofconsecutive blocks in scan order. An additional scan order other thanthe conventional slice raster scan order may be applied to the imagepartitioning. Section 5C illustrates examples C₀ and C₁ in whichscanning is performed clockwise or counter-clockwise with respect to astart block (Box-Out), and Section 5D illustrates examples D₀ and D₁ inwhich scanning is performed vertically with respect to a start block(Vertical). Information regarding the partitioning may includeinformation regarding the number of partitioning units, informationregarding the locations of the partitioning units (e.g., the first rankin scan order in the partitioning unit), information regarding the scanorder, and the like, and when the partitioning information is implicitlyincluded according to a predetermined rule, some of the partitioninginformation may not be generated.

In Section 5E, an image may be partitioned using lateral andlongitudinal partitioning lines. An existing tile may be partitioned bya lateral or longitudinal partitioning line. Thus, the partitioning maybe performed in the form of a quadrilateral space, but it may not bepossible to partition the image using the partitioning line. Forexample, an example in which an image is partitioned by somepartitioning lines for the image (e.g., a partitioning line between aleft boundary of E5 and a right boundary of E1, E3, and E4) may bepossible, and an example in which an image is partitioned by somepartitioning lines for the image (e.g., a partitioning line between alower boundary of E2 and E3 and an upper boundary of E4) may beimpossible. Also, the partitioning may be performed on a block unitbasis (e.g., after block partitioning is performed first) or may beperformed by the lateral or longitudinal partitioning line (e.g., thepartitioning is performed by the partitioning line, irrespective of theblock partitioning). Thus, each partitioning unit may not be a multipleof the block. Thus, partitioning information different from that of anexisting tile may be generated, and the partitioning information mayinclude information regarding the number of partitioning units,information regarding the locations of the partitioning units,information regarding the sizes of the partitioning units, etc. Forexample, the information regarding the locations of the partitioningunits may be generated as location information (e.g., which is measuredin pixel units or in block units) on the basis of a predeterminedlocation (e.g., at the upper left corner of the image), and theinformation regarding the sizes of the partitioning units may begenerated as information regarding the width and height of eachpartitioning unit (e.g., which is measured in pixel units or in blockunits).

Like the above example, the partitioning according to any user-definedsettings may be performed by applying a new partitioning method or bychanging some elements of the existing partitioning. That is, thepartitioning method may be supported by replacing or adding to aconventional partitioning method and may be supported by changing somesettings of the conventional partitioning method (a slice, a tile, etc.)(e.g., according to another scan order, by using another partitioningmethod in a quadrilateral shape to generate other partitioninginformation, or according to dependent encoding/decodingcharacteristics). Also, settings for configuring an additionalpartitioning unit (e.g., settings other than partitioning according to ascan order or partitioning according to a certain interval difference)may be supported, and an additional partitioning unit form (e.g., apolygonal form such as a triangle other than partitioning into aquadrilateral space) may be supported. Also, the image partitioningmethod may be supported on the basis of the type, characteristics, andthe like of an image. For example, a partial partitioning method (e.g.,the face of a 360-degree image) may be supported according to the type,characteristics, and the like of an image. Information regarding thepartitioning may be generated on the basis of the support.

In the image partitioning performing step, an image may be partitionedon the basis of identified partitioning type information. That is, theimage may be partitioned into a plurality of partitioning units on thebasis of the identified partitioning type and may be encoded or decodedon the basis of the acquired partitioning units.

In this case, whether to have encoding/decoding settings in eachpartitioning unit may be determined depending on the partitioning type.That is, setting information needed during an encoding/decoding processfor each partitioning unit may be assigned by an upper unit (e.g., apicture) or independent encoding/decoding settings may be provided foreach partitioning unit.

Generally, a slice may have independent encoding/decoding settings(e.g., a slice header) for each partitioning unit, and a tile cannothave independent encoding/decoding settings for each partitioning unitand may have settings dependent on encoding/decoding settings of apicture (e.g., PPS). In this case, information generated in associationwith a tile may be the partitioning information, and may be included inthe encoding/decoding settings of the picture. The present invention isnot limited to the above example, and modifications may be made thereto.

Encoding/decoding setting information for a tile may be generated inunits of videos, sequences, pictures, or the like. At least one piece ofencoding/decoding setting information is generated in an upper unit, andone piece of the generated encoding/decoding setting information may bereferenced. Alternatively, independent encoding/decoding settinginformation (e.g., a tile header) may be generated in tile units. Thisis different from the case of following one encoding/decoding settingdetermined in an upper unit in that encoding/decoding is performed whileat least one encoding/decoding setting is provided in tile units. Thatis, all the tiles may be encoded or decoded according to the sameencoding/decoding settings, or at least one tile may be encoded ordecoded according to different encoding/decoding settings from those ofthe other tiles.

The above example focuses on various encoding/decoding settings in thetile. However, the present invention is not limited thereto, and thesame or similar settings may be applied even to other partitioningtypes.

As an example, in some partitioning types, partitioning information maybe generated in an upper unit, and encoding or decoding may be performedaccording to a single encoding/decoding setting of the upper unit.

As an example, in some partitioning types, partitioning information maybe generated in an upper unit, and independent encoding/decodingsettings for each partitioning unit in the upper unit may be generated,and encoding or decoding may be performed according to the generatedencoding/decoding settings.

As an example, in some partitioning types, partitioning information maybe generated in an upper unit, and a plurality of pieces ofencoding/decoding setting information may be supported in the upperunit. Encoding or decoding may be performed according toencoding/decoding settings referenced by each partitioning unit.

As an example, in some partitioning types, partitioning information maybe generated in an upper unit, and independent encoding/decodingsettings may be generated in corresponding partitioning units. Encodingor decoding may be performed according to the generatedencoding/decoding settings.

As an example, in some partitioning types, independent encoding/decodingsettings including partitioning information may be generated incorresponding partitioning units, and encoding or decoding may beperformed according to the generated encoding/decoding settings.

Encoding/decoding setting information may include information needed toencode or decode a tile, such as a tile type, information regarding areferenced picture list, quantization parameter information,inter-prediction setting information, in-loop filtering settinginformation, in-loop filtering control information, a scan order,whether to perform encoding or decoding, and the like. Theencoding/decoding setting information may be used to explicitly generaterelated information or may have encoding/decoding settings that areimplicitly determined according to the format, characteristics, and thelike of the image which are determined in an upper unit. Also, therelated information may be explicitly generated on the basis ofinformation acquired through the settings.

Next, an example in which image partitioning is performed in theencoding/decoding apparatus according to an embodiment of the presentinvention will be described.

A partitioning process may be performed on an input image beforeencoding is started. The image may be partitioned using the partitioninginformation (e.g., image partitioning information, partitioning unitsetting information, etc.) and then may be encoded in partitioningunits. The image encoding data may be stored in the memory after theencoding is complete, and may be added to a bitstream and thentransmitted.

A partitioning process may be performed before decoding is started. Theimage may be partitioned using the partitioning information (e.g., imagepartitioning information, partitioning unit setting information, etc.),and then image decoding data may be parsed and decoded in partitioningunits. The image decoding data may be stored in the memory after thedecoding is complete, and the plurality of partitioning units are mergedinto a single unit, and thus an image may be output.

Through the above example, the image partitioning process has beendescribed. Also, according to the present invention, a plurality ofpartitioning processes may be performed.

For example, an image may be partitioned, and partitioning units of theimage may be partitioned. The partitioning may be the same partitioningprocess (e.g., slice/slice, tile/tile, etc.) or a different partitioningprocess (e.g., slice/tile, tile/slice, tile/face, face/title,slice/face, face/slice, etc.). In this case, the following partitioningprocess may be performed on the basis of the preceding partitioningresult, and partitioning information generated during the followingpartitioning process may be generated on the basis of a precedingpartitioning result.

Also, a plurality of partitioning processes A may be performed, and thepartitioning process may be a different partitioning process (e.g.,slice/face, tile/face, and the like). In this case, the followingpartitioning process may be performed on the basis of or independentlyof the preceding partitioning result, and partitioning informationgenerated during the following partitioning process may be generated onthe basis of or independently of the preceding partitioning result.

The plurality of image partitioning processes may be determinedaccording to encoding/decoding settings. However, the present inventionis not limited to the above example, and various modifications may bemade thereto.

The encoder may add the information generated during the above processto a bitstream in units of at least one of sequences, pictures, slices,tiles, and the like, and the decoder may parse related information fromthe bitstream. That is, the information may be added to one unit and maybe duplicated and added to a plurality of units. For example, a syntaxelement indicating whether to support some information or a syntaxelement indicating whether to perform activation may be generated insome units (e.g., an upper unit), and the same or similar informationmay be generated in some units (e.g., a lower unit). That is, even whenrelated information is supported and set in the upper unit, the lowerunit may have individual settings. This description is not limited tothe above example and may be applied in common to the present invention.Also, the information may be included in the bitstream in the form ofSEI or metadata.

Generally, an input image may be encoded or decoded as it is, but theencoding or decoding may be performed after the image is resized(expanded or reduced; resolution adjustment). For example, in ahierarchical coding scheme (Scalability Video Coding) for supportingspatial, temporal, and image-quality scalability, image resizing such asthe entire expansion and reduction of an image may be performed.Alternatively, the image resizing may be performed such as partialexpansion and reduction of an image. The image resizing may be variouslyperformed, that is, may be performed for the purpose of adaptability toencoding environments, for the purpose of encoding uniformity, for thepurpose of encoding efficiency, for the purpose of image qualityimprovement, or according to the type, characteristics, and the like ofan image.

As a first example, the resizing process may be performed during aprocess performed according to the characteristics, type, and the likeof an image (e.g., hierarchical encoding, 360-degree image encoding,etc.).

As a second example, the resizing process may be performed at an initialencoding/decoding step. The resizing process may be performed beforeencoding or decoding is performed. The resized image may be encoded ordecoded.

As a third example, the resizing process may be performed during aprediction step (intra-prediction or inter-prediction) or beforeprediction. During the resizing process, image information (e.g.,information regarding a pixel referenced for intra-prediction,information regarding an intra-prediction mode, information regardingreference pictures used for inter-prediction, information regarding aninter-prediction prediction mode, etc.) may be used at the predictionstep.

As a fourth example, the resizing process may be performed during afiltering step or before filtering. In the resizing process, imageinformation in the filtering step may be used (e.g., pixel informationto be applied to the deblocking filter, pixel information to be appliedto SAO, information regarding SAO filtering, pixel information appliedto ALF, information regarding ALF filtering, and the like).

Also, after the resizing process is performed, the image may beprocessed through an inverse resizing process and changed to an imagebefore resizing (in terms of an image size) or may be unchanged. Thismay be determined according to encoding/decoding settings (e.g.,characteristics in which the resizing is performed). In this case, theresizing process may be an expansion process while the inverse resizingprocess is a reduction process and may be a reduction process while theinverse resizing process is an expansion process.

When the resizing process is performed according to the first to fourthexamples, the inverse resizing process is performed in the followingstep so that an image before resizing may be acquired.

When the resizing process is performed through hierarchical encoding oraccording to the third example (or when a reference picture is resizedin inter-prediction), the inverse resizing process may not be performedin the following step.

In an embodiment of the present invention, the image resizing processmay be performed solely or along with the inverse process. The followingexample description will focus on the resizing process. In this case,since the inverse resizing process is an inverse process for theresizing process, a description of the inverse resizing process will beomitted in order to prevent redundant descriptions. However, it isobvious that those skilled in the art can recognize the same things asdescribed literally.

FIG. 6 is an example diagram of a general image resizing method.

Referring to Section 6A, an expanded image P₀+P₁ may be acquired byadding a specific region P₁ to an initial image P₀ (or an image beforeresizing; which is indicated by a thick solid line).

Referring to Section 6B, a reduced image S₀ may be acquired by removinga specific region S₁ from an initial image S₀+S₁.

Referring to Section 6C, a resized image T₀+T₁ may be acquired by addinga specific region T₁ to an initial image T₀+T₂ and removing a specificregion T₂ from the entire image.

According to the present invention, the following description focuses ona resizing process for expansion and a resizing process for reduction.However, the present invention is not limited thereto, and it should beunderstood to include a case in which expansion and reduction areapplied in combination, as shown in Section 6C.

FIG. 7 is an example diagram of image resizing according to anembodiment of the present invention.

During the resizing process, an image expansion method will be describedwith reference to Section 7A, and an image reduction method will bedescribed with reference to Section 7B.

In Section 7A, an image before resizing is S0, and an image afterresizing is S1. In Section 7B, an image before resizing is T0, and animage after resizing is T1.

When an image is expanded as shown in Section 7A, the image may beexpanded in an “up” direction ET, a “down” direction EL, a “left”direction EB, or a “right” direction ER. When an image is reduced asshown in Section 7B, the image may be reduced in an “up” direction RT, a“down” direction RL, a “left” direction RB, or a “right” direction RR.

Comparing the image expansion and the image reduction, the “up”direction, the “down” direction, the “left” direction, and the “right”direction of the expansion may correspond to the “down” direction, the“up” direction, the “right” direction, and the “left” direction of thereduction. Thus, the following description focuses on the imageexpansion, but it should be understood that a description of the imagereduction is included.

In the following description, the image expansion or reduction isperformed in the “up” direction, the “down” direction, the “left”direction, and the “right” direction. However, it should be alsounderstood that the resizing may be performed in an “up and left”direction, an “up and right” direction, a “down and left” direction, ora “down and right” direction.

In this case, when the expansion is performed in the “down and right”direction, regions RC and BC are acquired, and a region BR may or maynot be acquired according to encoding/decoding settings. That is,regions TL, TR, BL, and BR may or may not be acquired, but forconvenience of description, corner regions (i.e., the regions TL, TR,BL, and BR) will be described as capable of being acquired.

The image resizing process according to an embodiment of the presentinvention may be performed in at least one direction. For example, theimage resizing process may be performed in all directions such as up,down, left, and right, may be performed in two or more directionsselected from up, down, left, and right (left+right, up+down, up+left,up+right, down+left, down+right, up+left+right, down+left+right,up+down+left, up+down+right, etc.), or may be performed in only onedirection selected from up, down, left, and right.

For example, the resizing may be performed in a “left+right” direction,an “up+down” direction, a “left and up+right and down” direction, and a“left and down+right and up” direction, which are symmetricallyexpandable to both ends with respect to the center of an image, may beperformed in a “left+right” direction, a “left and up+right up”direction, and a “left and down+right and down” direction, which arevertically symmetrically expandable with respect to the image, and maybe performed in an “up+down” direction, a “left and up+left and down”direction, and a “right and up+right and down” direction, which arehorizontally symmetrically expandable with respect to the image. Otherresizing may be performed.

In Sections 7A and 7B, the size of the image before resizing S0 or TO isdefined as P_Width×P_Height, and the size of the image after resizing S1or T1 is defined as P′_Width×P′_Height. Here, when resizing values inthe “left” direction, the “right” direction, the “up” direction, and the“down” direction are defined as Var_L, Var_R, Var_T, and Var_B (orcollectively defined as Var_x), the size of the image after resizing maybe expressed as (P_Width+Var_L+Var_R)×(P_Height+Var_T+Var_B). In thiscase, Var_L, Var_R, Var_T, and Var_B, which are the resizing values inthe “left” direction, the “right” direction, the “up” direction, and the“down” direction, may be Exp_L, Exp_R, Exp_T, and Exp_B (here, Exp_x ispositive) for the image expansion (in Section 7A) and may be −Rec_L,−Rec_R, −Rec_T, and −Rec_B for the image reduction (which is representedas negative values for the image reduction when Rec_L, Rec_R, Rec_T, andRec_B are defined as positive values). Also, upper left-handcoordinates, upper right-hand coordinates, lower left-hand coordinates,and lower right-hand coordinates of the image before resizing may be(0,0), (P_Width−1,0), (0,P_Height−1), and (P_Width−1,P_Height−1), andthe upper left-hand coordinates, upper right-hand coordinates, lowerleft-hand coordinates, and lower right-hand coordinates of the imageafter resizing may be represented as (0,0), (P′_Width−1,0),(0,P′_Height−1), and (P′_Width−1,P′_Height−1). The size of the region(here, TL to BR; i is an index for identifying TL to BR) that is changed(or acquired or removed) through the resizing may be M[i]×N[i] and maybe represented as Var_X×Var_Y (this example assumes that X is L or R andY is T or B). M and N may have various values and may have the samesettings irrespective of i or may have individual settings according toi. The various examples will be described below.

Referring to Section 7A, S1 may be configured to include some or all ofthe regions TL to BR (upper left to lower right), which are to begenerated through expansion on S0 in several directions. Referring toSection 7B, T1 may be configured to exclude, from T0, all or some of theregions TL to BR, which are to be removed through reduction in severaldirections.

In Section 7A, when an existing image S0 is expanded in an “up”direction, a “down” direction, a “left” direction, and a “right”direction, the image may include the regions TC, BC, LC, and RC acquiredthrough the resizing processes and may further include the regions TL,TR, BL, and BR.

As an example, when the expansion is performed in the “up” direction ET,the image may be constructed by adding the region TC to the existingimage S0 and may include the region TL or TR along with expansion in atleast one different direction EL or ER.

As an example, when the expansion is performed in the “down” directionEB, the image may be constructed by adding the region BC to the existingimage S0 and may include the region BL or BR along with expansion in atleast one different direction EL or ER.

As an example, when the expansion is performed in the “left” directionEL, the image may be constructed by adding the region LC to the existingimage S0 and may include the region TL or BL along with expansion in atleast one different direction ET or EB.

As an example, when the expansion is performed in the “right” directionER, the image may be constructed by adding the region RC to the existingimage S0 and may include the region TR or BR along with expansion in atleast one different direction ET or EB.

According to an embodiment of the present invention, it is possible toprovide settings (e.g., spa_ref_enabled_flag or tem_ref_enabled_flag)for spatially or temporally limiting referenceability of the resizedregion (this example assumes expansion).

That is, reference to data of the region that is spatially or temporallyresized according to encoding/decoding settings may be allowed (e.g.,spa_ref_enabled_flag=1 or tem_ref_enabled_flag=1) or limited (e.g.,spa_ref_enabled_flag=0 or tem_ref_enabled_flag=0).

The encoding/decoding of the images S0 and T1 before resizing and theregions TC, BC, LC, RC, TL, TR, BL, and BR added or deleted duringresizing may be performed as follows.

For example, when the image before resizing and the added or deletedregion are encoded or decoded, the data regarding the image beforeresizing and the data regarding the added or deleted region (data afterthe encoding or decoding is complete; a pixel value orprediction-related information) may be spatially or temporallyreferenced to each other.

Alternatively, the image before resizing and the data regarding theadded or deleted region may be spatially referenced while the dataregarding the image before resizing may be temporally referenced and thedata regarding the added or deleted region cannot be temporallyreferenced.

That is, it is possible to provide settings for limitingreferenceability of the added or deleted region. The setting informationregarding the referenceability of the added or deleted region may beexplicitly generated or implicitly determined.

The image resizing process according to an embodiment of the presentinvention may include an image resizing indication step, an imageresizing type identification step, and/or an image resizing performingstep. Also, the image encoding apparatus and the image decodingapparatus may include an image resizing indication part, an imageresizing type identification part, and an image resizing performingpart, which are configured to perform the image resizing indicationstep, the image resizing type identification step, and the imageresizing performing step, respectively. For the encoding, a relatedsyntax element may be generated. For the decoding, a related syntaxelement may be parsed.

In the image resizing indication step, whether to perform image resizingmay be determined. For example, when a signal indicating image resizing(e.g., img_resizing_enabled_flag) is confirmed, the resizing may beperformed. When the signal indicating image resizing is not confirmed,the resizing may not be performed, or the resizing may be performed byconfirming other encoding/decoding information. Also, although thesignal indicating image resizing is not provided, the signal indicatingimage resizing may be implicitly activated or deactivated according toencoding/decoding settings (e.g., the characteristics, type, and thelike of an image). When the resizing is performed, correspondingresizing-related information may be generated or may be implicitlydetermined.

When the signal indicating image resizing is provided, the correspondingsignal is a signal for indicating whether to perform the image resizing.Whether to resize the corresponding image may be determined according tothe signal.

For example, it is assumed that a signal indicating image resizing(e.g., img_resizing_enabled_flag) is confirmed. When the correspondingsignal is activated (e.g., img_resizing_enabled_flag=1), the imageresizing may be performed. When the corresponding signal is deactivated(e.g., img_resizing_enabled_flag=0), the image resizing may not beperformed.

Also, when the signal indicating image resizing is not provided, theresizing may not be performed, or whether to resize a correspondingimage may be determined by another signal.

For example, when an input image is partitioned in block units, theresizing may be performed according to whether the size (e.g., the widthor height) of the image is a multiple of the size (e.g., the width orheight) of the block (for expansion in this example, it is assumed thatthe resizing process is performed when the image size is not a multipleof the block size). That is, when the width of the image is not amultiple of the width of the block or when the height of the image isnot a multiple of the height of the block, the resizing may beperformed. In this case, the resizing information (e.g., a resizingdirection, a resizing value, etc.) may be determined according to theencoding/decoding information (e.g., the size of the image, the size ofthe block, etc.). Alternatively, the resizing may be performed accordingto the characteristics, type (e.g., a 360-degree image), and the like ofthe image, and the resizing information may be explicitly generated ormay be assigned as a predetermined value. The present invention is notlimited to the above example, and modifications may be made thereto.

In the image resizing type identification step, an image resizing typemay be identified. The image resizing type may be defined by a resizingmethod, resizing information, and the like. For example, scalefactor-based resizing, offset factor-based resizing, and the like may beperformed. The present invention is not limited to the above example,and the methods may be applied in combination. For convenience ofdescription, the following description will focus on the scalefactor-based resizing and the offset factor-based resizing.

For the scale factor, the resizing may be performed by multiplication ordivision based on the size of the image. Information regarding theresizing operations (e.g., expansion or reduction) may be explicitlygenerated, and the expansion or reduction process may be performedaccording to the corresponding information. Also, the resizing processmay be performed as a predetermined operation (e.g., one of theexpansion operation and the reduction operation) according toencoding/decoding settings. In this case, the information regarding theresizing operations will be omitted. For example, when the imageresizing is activated in the image resizing indication step, the imageresizing may be performed as a predetermined operation.

The resizing direction may be at least one direction selected from up,down, left, and right. At least one scale factor may be requireddepending on the resizing direction. That is, one scale factor (here,unidirectional) may be required for each direction, one scale factor(here, bidirectional) may be required for a lateral or longitudinaldirection, and one scale factor (here, omnidirectional) may be requiredfor all directions of the image. Also, the resizing direction is notlimited to the above example, and modifications may be made thereto.

The scale factor may have a positive value and may have rangeinformation differing depending on encoding/decoding settings. Forexample, when information is generated by combining the resizingoperation and the scale factor, the scale factor may be used as amultiplicand. The scale factor being greater than 0 or less than 1 maymean a reduction operation, the scale factor being greater than 1 maymean an expansion operation, and the scale factor being 1 may mean thatthe resizing is not performed. As another example, when scale factorinformation is generated irrespective of the resizing operation, thescale factor for the expansion operation may be used as a multiplicand,and the scale factor for the reduction operation may be used as adividend.

A process of changing images before resizing S0 and T0 to images afterresizing (here, S1 and T1) will be described with reference to Sections7A and 7B of FIG. 7 again.

As an example, when one scale factor (referred to as sc) is used in allthe directions of the image and the resizing direction is a “down+right”direction, the resizing directions are ER and EB (or RR and RB), theresizing values Var_L(Exp_L or Rec_L) and Var_T(Exp_T or Rec_T) are 0,and Var_R(Exp_R or Rec_R) and Var_B(Exp_B or Rec_B) may be expressed asP_Width×(sc−1) and P_Height×(sc−1). Accordingly, the image afterresizing may be (P_Width×sc)×(P_Height×sc).

As an example, when respective scale factors (here, sc_w and sc_h) areused in a lateral direction or a longitudinal direction of the image andthe resizing directions are a “left+right” direction and an “up+down”direction (up+down+left+right when two are operated), the resizingdirection may be ET, EB, EL, and ER, the resizing values Var_T and Var_Bmay be P_Height×(sc_h−1)/2, and Var_L and Var_R may beP_Width×(sc_w−1)/2. Accordingly, the image after resizing may be(P_Width×sc_w)×(P_Height×sc_h).

For the offset factor, the resizing may be performed by addition orsubtraction based on the size of the image. Alternatively, the resizingmay be performed by addition or subtraction based on encoding/decodinginformation of the image. Alternatively, the resizing may be performedby independent addition or subtraction. That is, the resizing processmay have dependent or independent settings.

Information regarding the resizing operations (e.g., expansion orreduction) may be explicitly generated, and the expansion or reductionprocess may be performed according to the corresponding information.Also, the resizing operations may be performed as a predeterminedoperation (e.g., one of the expansion operation and the reductionoperation) according to encoding/decoding settings. In this case, theinformation regarding the resizing operations may be omitted. Forexample, when the image resizing is activated in the image resizingindication step, the image resizing may be performed as a predeterminedoperation.

The resizing direction may be at least one direction selected from up,down, left, and right. At least one offset factor may be requireddepending on the resizing direction. That is, one offset factor (here,unidirectional) may be required for each direction, one offset factor(here, symmetrically bidirectional) may be required for a lateraldirection or longitudinal direction, one offset factor (here,asymmetrically bidirectional) may be required according to a partialcombination of the directions, and one offset factor (here,omnidirectional) may be required for all directions of the image. Also,the resizing direction is not limited to the above example, andmodifications may be made thereto.

The offset factor may have a positive value or have both a positivevalue and a negative value, and may have range information differingdepending on encoding/decoding settings. For example, when informationis generated in combination of the resizing operation and the offsetfactor (here, it is assumed that the offset factor has both a positivevalue and a negative value), the offset factor may be used as a value tobe added or subtracted depending on sign information of the offsetfactor. The offset factor being greater than 0 may mean an expansionoperation, the offset factor being less than 0 may mean a reductionoperation, and the offset factor being 0 may mean that the resizing isnot performed. As another example, when offset factor information isgenerated separately from the resizing operation (here, it is assumedthat the offset factor has a positive value), the offset factor may beused as a value to be added or subtracted depending on the resizingoperation. The offset factor being greater than 0 may mean that theexpansion or reduction operation may be performed depending on theresizing operation, and the offset factor being 0 may mean that theresizing is not performed.

A method of changing images before resizing S0 and T0 to images afterresizing S1 and T1 using an offset factor will be described withreference to Sections 7A and 7B of FIG. 7 again.

As an example, when one offset factor (referred to as os) is used in allthe directions of the image and the resizing direction is an“up+down+left+right” direction, the resizing directions may be ET, EB,EL, and ER (or RT, RB, RL, and RR), and the resizing values Var_T,Var_B, Var_L, and Var_R may be os. The size of the image after resizingmay be (P_Width+os)×(P_Height+os).

As an example, when an offset factor os_w or os_h is used in a lateralor longitudinal direction of the image and the resizing directions are a“left+right” direction and an “up+down” direction (an“up+down+left+right” direction when two are operated), the resizingdirections may be ET, EB, EL, and ER (or RT, RB, RL, and RR), theresizing values Var_T and Var_B may be os_h, and the resizing valuesVar_L and Var_R may be os_w. The size of the image after resizing may be{P_Width+(os_w×2)}×{P_Height+(os_h×2)}.

As an example, when the resizing directions are a “down” direction and a“right” direction (a “down+right” direction when being operatedtogether) and an offset factor os_b or os_r is used depending on theresizing direction, the resizing directions may be EB and ER (or RB andRR), the resizing value Var_B may be os_b, and the resizing value Var_Rmay be os_r. The size of the image after resizing may be(P_Width+os_r)×(P_Height+os_b).

As an example, when the offset factor os_t, os_b, os_l, or os_r is useddepending on the direction of the image and the resizing directions arean “up” direction, a “down” direction, a “left” direction, and a “right”direction (an “up+down+left+right” direction when all are operated), theresizing directions may be ET, EB, EL, and ER (or RT, RB, RL, and RR),the resizing value Var_T may be os_t, the resizing value Var_B may beos_b, the resizing value Var_L may be os_l, and the resizing value Var_Rmay be os_r. The size of the image after resizing may be(P_Width+os_l+os_r)×(P_Height+os_t+os_b).

The above example indicates a case in which the offset factor is used asa resizing value Var_T, Var_B, Var_L, or Var_R during the resizingprocess. That is, this means that the offset factor is used as theresizing value without any change, which may be an example of theresizing that is independently performed. Alternatively, the offsetfactor may be used as an input variable of the resizing value. Indetail, the offset factor may be assigned as an input variable, and theresizing value may be acquired through a series of processes accordingto encoding/decoding settings, which may be an example of the resizingthat is performed on the basis of predetermined information (e.g., animage size, encoding/decoding information, etc.) or an example of theresizing that is dependently performed.

For example, the offset factor may be a multiple (e.g., 1, 2, 4, 6, 8,and 16) or an exponent (e.g., exponents of 2, such as 1, 2, 4, 8, 16,32, 64, 128, and 256) of a predetermined value (here, an integer).Alternatively, the offset factor may be a multiple or an exponent of avalue acquired based on encoding/decoding settings (e.g., a value thatis set based on a motion search range of inter-prediction).Alternatively, the offset factor may be a multiple or an integer of aunit (here, assuming A×B) that is acquired from the picture partitioningpart. Alternatively, the offset factor may be a multiple of a unit(here, assuming E×F such as a tile) that is acquired from the picturepartitioning part.

Alternatively, the offset factor may be a value that is less than orequal to the width and height of the unit acquired from the picturepartitioning part. In the above example, the multiple or the exponentmay have a value of 1. However, the present invention is not limited tothe above example, and modifications may be made thereto. For example,when the offset factor is n, Var_x may be 2×n or 2^(n).

Also, individual offset factors may be supported according to colorcomponents. Offset factors for some color components may be supported,and thus offset factor information for other color components may bederived. For example, when an offset factor A for the luminancecomponent (here, assuming that a composition ratio of the luminancecomponent with respect to the chrominance component is 2:1) isexplicitly generated, an offset factor A/2 for the chrominance componentmay be implicitly acquired. Alternatively, when the offset factor A forthe chrominance component is explicitly generated, the offset factor 2Afor the luminance component may be implicitly acquired.

Information regarding the resizing direction and the resizing value maybe explicitly generated, and the resizing process may be performedaccording to the corresponding information. Also, the information may beimplicitly determined according to encoding/decoding settings, and theresizing process may be performed according to the determinedinformation. At least one predetermined direction or resizing value maybe assigned, and in this case, the related information may be omitted.In this case, the encoding/decoding settings may be determined on thebasis of the characteristics, type, encoding information, and the likeof an image. For example, at least one resizing direction may bepredetermined according to at least one resizing operation, at least oneresizing value may be predetermined according to at least one resizingoperation, and at least one resizing value may be predeterminedaccording to at least one resizing direction. Also, the resizingdirection, the resizing value, and the like during the inverse resizingprocess may be derived from the resizing direction, the resizing value,and the like which are applied during the resizing process. In thiscase, the resizing value that is implicitly determined may be one of theabove examples (examples in which the resizing value is variouslyacquired).

Also, the multiplication or division has been described in the aboveexample, but a shift operation may be used depending on theimplementation of the encoder/decoder. The multiplication may beimplemented through a left shift operation, and the division may beimplemented through a right shift operation. This description is notlimited to the above example and may be applied in common to the presentinvention.

In the image resizing performing step, image resizing may be performedon the basis of identified resizing information. That is, the imageresizing may be performed on the basis of information regarding aresizing type, a resizing operation, a resizing direction, a resizingvalue, and the like, and encoding/decoding may be performed on the basisof an acquired image after resizing.

Also, in the image resizing performing step, the resizing may beperformed using at least one data processing method. In detail, theresizing may be performed on a region to be resized according to theresizing type and the resizing operation by using at least one dataprocessing method. For example, depending on the resizing type, how tofill data may be determined when the resizing is for expansion, and howto remove data may be determined when the resizing is for reduction.

In summary, in the image resizing performing step, the image resizingmay be performed on the basis of identified resizing information.Alternatively, in the image resizing performing step, the image resizingmay be performed on the basis of the resizing information and a dataprocessing method. The above two cases may differ from each other inthat only the size of an image to be encoded or decoded is adjusted orin that even data processing for the image size and for the region to beresized is considered. In the image resizing performing step, whether toperform the data processing method may be determined depending on astep, a position, and the like in which the resizing process is applied.The following description focuses on an example in which the resizing isperformed on the basis of the data processing method, but the presentinvention is not limited thereto.

When the offset factor-based resizing is performed, the resizing for theexpansion and the resizing for the reduction may be performed usingvarious methods. For the expansion, the resizing may be performed usingat least one data filling method. For the reduction, the resizing may beperformed using at least one data removal method. In this case, when theoffset factor-based resizing is performed, the resized region(expansion) may be filled with new data or original image data directlyor after modification, and the resized region (reduction) may be removedsimply or through a series of processes.

When the scale factor-based resizing is performed, in some cases (e.g.,hierarchical encoding), the resizing for the expansion may be performedby applying up-sampling, and the resizing for the reduction may beperformed by applying down-sampling. For example, at least oneup-sampling filter may be used for the expansion, and at least onedown-sampling filter may be used for the reduction. A horizontallyapplied filter may be the same as or different from a vertically appliedfilter. In this case, when the scale factor-based resizing is performed,new data is neither generated in nor removed from the resized region,but original image data may be rearranged using a method such asinterpolation. A data processing method associated with the resizing maybe classified according to a filter used for the sampling. Also, in somecases (e.g., a case similar to that of the offset factor), the resizingfor the expansion may be performed using a method of filling at leastone piece of data, and the resizing for the reduction may be performedusing a method of removing at least one piece of data. According to thepresent invention, the following description focuses on the dataprocessing method corresponding to when the offset factor-based resizingis performed.

Generally, a predetermined data processing method may be used in theregion to be resized, but at least one data processing method may beused in the region to be resized as in the following example. Selectioninformation for the data processing method may be generated. The formermay mean that the resizing is performed through a fixed data processingmethod, and the latter may mean that the resizing is performed throughan adaptive data processing method.

Also, a data processing method may be applied to all (TL, TC, TR, . . ., BR in Sections 7A and 7B) or some (e.g., each or a combination of TLto BR in Sections 7A and 7B) of the regions to be added or deletedduring the resizing.

FIG. 8 is an example diagram of a method of constructing a regiongenerated through expansion in the image resizing method according to anembodiment of the present invention.

Referring to Section 8A, for convenience of description, an image may bepartitioned into regions TL, TC, TR, LC, C, RC, BL, BC, and BR, whichcorrespond to an upper left position, an upper position, an upper rightposition, a left position, a center position, a right position, a lowerleft position, a lower position, and a lower right position of theimage. In the following description, the image is expanded in a“down+right” direction, but it should be understood that the descriptionmay be applied to the other expansion directions.

A region added according to the expansion of the image may beconstructed using various methods. For example, the region may be filledwith an arbitrary value or may be filled with reference to some data ofthe image.

Referring to Section 8B, generated regions A₀ and A₂ may be filled withan arbitrary pixel value. The arbitrary pixel value may be determinedusing various methods.

As an example, the arbitrary pixel value may be one pixel in a pixelvalue range (e.g., from 0 to 1<<(bit_depth)−1) which may be expressedusing a bit depth. For example, the arbitrary pixel value may be aminimum, a maximum, a median (e.g., 1<<(bit_depth−1), etc.), or the likein the pixel value range (here, bit_depth indicates a bit depth).

As an example, the arbitrary pixel value may be one pixel in the pixelvalue range (e.g., from min_(P) to max_(P); min_(P) and max_(P) indicatea minimum value and a maximum value among the pixels belonging to theimage; min_(P) is greater than or equal to 0; max_(P) is smaller than orequal to 1<<(bit_depth)−1) of the pixels belonging to the image. Forexample, the arbitrary pixel value may be a minimum, a maximum, amedian, an average (of at least two pixels), a weighted sum, etc. of thepixel value range.

As an example, the arbitrary pixel value may be a value that isdetermined in a pixel value range belonging to the specific regionincluded in the image. For example, when A₀ is constructed, the specificregion may be TR+RC+BR. Also, the specific region may be provided as aregion corresponding to 3×9 of TR, RC, and BR or a region correspondingto 1×9<which is assumed as the rightmost line>. This may depend onencoding/decoding settings. In this case, the specific region may be aunit to be partitioned by the picture partitioning part. In detail, thearbitrary pixel value may be a minimum, a maximum, a median, an average(of at least two pixels), a weighted sum, etc. of the pixel value range.

Referring to Section 8B again, a region A₁ to be added along with imageexpansion may be filled with pattern information (e.g., the pattern isassumed as using a plurality of pixels; there is no need to followcertain rules) which is generated using a plurality of pixel values. Inthis case, the pattern information may be defined according toencoding/decoding settings or related information may be generated. Thegenerated region may be filled with at least one piece of patterninformation.

Referring to Section 8C, a region added along with the image expansionmay be constructed with reference to pixels of the specific regionincluded in the image. In detail, the added region may be constructed bycopying or padding pixels (hereinafter referred to as reference pixels)in a region adjacent to the added region. In this case, the pixels inthe region adjacent to the added region may be a pixel before encodingor a pixel after encoding (or decoding). For example, the referencepixel may refer to a pixel of an input image when the resizing isperformed in a pre-encoding step, and the reference pixel may refer to apixel of a restored image when the resizing is performed in anintra-prediction reference pixel generation step, a reference picturegeneration step, a filtering step, and the like. In this example, it isassumed that the nearest pixel is used in the added region, but thepresent invention is not limited thereto.

A region A₀, which is generated when the image is expanded leftward orrightward in association with lateral image resizing, may be constructedby horizontally padding (Z0) outer pixels adjacent to the generatedregion A₀, and a region A₁, which is generated when the image isexpanded upward or downward in association with longitudinal imageresizing, may be constructed by vertically padding (Z1) outer pixelsadjacent to the generated region A₁. Also, a region A₂, which isgenerated when the image is expanded downward and rightward, may beconstructed by diagonally padding (Z2) outer pixels adjacent to thegenerated region A₂.

Referring to Section 8D, generated regions B′0 to B′2 may be constructedwith reference to data of specific regions B0 to B2 included in theimage. In Section 8D, unlike Section 8C, a region that is not adjacentto the generated region may be referenced.

For example, when a region having high correlation with the generatedregion is present in the image, the generated region may be filled withreference to pixels of the region having high correlation. In this case,the location information, size information, etc. of the region havinghigh correlation may be generated. Alternatively, when the region havinghigh correlation is present through encoding/decoding information of thecharacteristics, type, and the like of the image, and the locationinformation, the size information, and the like of the region havinghigh correlation may be implicitly checked (e.g., as for a 360-degreeimage), the generated region may be filled with data of thecorresponding region. In this case, the location information, sizeinformation, etc. of the corresponding region may be omitted.

As an example, a region B′2, which is generated when the image isexpanded leftward or rightward in association with lateral imageresizing, may be filled with reference to pixels in a region B2 oppositeto the region generated when the image is expanded leftward or rightwardin association with the lateral resizing.

As an example, a region B1, which is generated when the image isexpanded upward or downward in association with longitudinal imageresizing, may be filled with reference to pixels in a region B1 oppositeto the region generated when the image is expanded upward or downward inassociation with the longitudinal resizing.

As an example, a region B′0, which is generated when the image isexpanded through some image resizing (here, diagonally with respect tothe image center), may be filled with reference to pixels in a region B0or TL opposite to the generated region.

An example in which continuity is present at a boundary between bothends of the image and in which data of a region symmetric with respectto the resizing direction is acquired has been described. However, thepresent invention is not limited thereto, and data of other regions TLto BR may be acquired.

When the generated region is filled with data of a specific region ofthe image, the data of the corresponding region may be copied and usedto fill the generated region as it is, or the data of the correspondingregion may be transformed on the basis of the characteristics, type, andthe like of the image and used to fill the generated region. In thiscase, copying the data as it is may mean that the pixel value of thecorresponding region is used without any change, and performing thetransformation process may mean that the pixel value of thecorresponding region is not used without any change. That is, at leastone pixel value of the corresponding region may be changed through thetransformation process. The generated region may be filled with thechanged pixel value, or at least one of locations at which some pixelsare acquired may differ from the other locations. That is, in order tofill the generated region of A×B, C×D data other than A×B data, of thecorresponding region may be used. In other words, at least one of motionvectors applied to the pixels with which the generated region is filledmay differ from the other pixels. In the above example, when a360-degree image is composed of a plurality of faces according to aprojection format, the generated region may be filled with data of theother faces. A data processing method for filling a region generatedwhen the image is expanded through image resizing is not limited to theabove example. The data processing method may be improved or changed, oran additional data processing method may be used.

A plurality of candidate groups for the data processing method may besupported according to encoding/decoding settings, and informationregarding selection of a data processing method from among the pluralityof candidate groups may be generated and added to a bitstream. Forexample, one data processing method may be selected from among a fillingmethod by using a predetermined pixel value, a filling method by copyingouter pixels, a filling method by copying a specific region of an image,a filling method by transforming a specific region of an image, and thelike, and related selection information may be generated. Also, the dataprocessing method may be implicitly determined.

For example, a data processing method applied to all the regions (here,the regions TL to BR in Section 7A), which are to be generated alongwith expansion through image resizing, may be one of the a fillingmethod by using a predetermined pixel value, the filling method bycopying outer pixels, the filling method by copying a specific region ofan image, the filling method by transforming a specific region of animage, and the like, and related selection information may be generated.Also, one predetermined data processing method applied to the entireregion may be determined.

Alternatively, a data processing method applied to the regions (here,each of or two or more of the regions TL to BR in Section 7A of FIG. 7), which are to be generated along with expansion through imageresizing, may be one of the filling method by using a predeterminedpixel value, the filling method by copying outer pixels, the fillingmethod by copying a specific region of an image, the filling method bytransforming a specific region of an image, and the like, and relatedselection information may be generated. Also, one predetermined dataprocessing method applied to at least one region may be determined.

FIG. 9 is an example diagram of a method of constructing a region to bedeleted through reduction and a region to be generated in the imageresizing method according to an embodiment of the present invention.

The region to be deleted in the image reduction process may be removednot only simply but also after a series of application processes.

Referring to Section 9A, during the image reduction process, specificregions A₀, A₁, and A₂ may be simply removed without an additionalapplication process. In this case, an image A may be partitioned intoregions TL to BR, as shown in Section 8A.

Referring to Section 9B, the regions A₀ to A₂ may be removed and may beutilized as reference information when the image A is encoded ordecoded. For example, the deleted regions A₀ to A₂ may be utilizedduring a process of restoring or correcting specific regions of theimage A that are deleted through reduction. During the restoration orcorrection process, a weighted sum, an average, and the like of tworegions (a deleted region and a generated region) may be used. Also, therestoration or correction process may be a process that may be appliedwhen the two regions have high correlation.

As an example, a region B′2, which is deleted when the image is reducedleftward or rightward in association with lateral image resizing, may beused to restore or correct pixels in a region B2, LC opposite to theregion deleted when the image is reduced leftward or rightward inassociation with the lateral resizing, and then may be removed from thememory.

As an example, a region B′1, which is deleted when the image is reducedupward or downward in association with longitudinal image resizing, maybe used for an encoding/decoding process (a restoration or correctionprocess) of a region B1, TR opposite to the region deleted when theimage is reduced upward or downward in association with the longitudinalresizing, and then may be removed from the memory.

As an example, a region B′0, which is deleted when the image is reducedthrough some image resizing (here, diagonally with respect to the imagecenter), may be used for an encoding/decoding process (a restoration orcorrection process) of a region B0 or TL opposite to the deleted region,and then may be removed from the memory.

An example in which continuity is present at a boundary between bothends of the image and in which data of a region symmetric with respectto the resizing direction is used for the restoration or correction hasbeen described. However, the present invention is not limited thereto,and data of regions TL to BR other than the symmetric region may be usedfor the restoration or correction and then may be removed from thememory.

A data processing method for removing a region to be deleted is notlimited to the above example. The data processing method may be improvedor changed, or an additional data processing method may be used.

A plurality of candidate groups for the data processing method may besupported according to encoding/decoding settings, and related selectioninformation may be generated and added to a bitstream. For example, onedata processing method may be selected from among a method of simplyremoving a region to be deleted, a method of removing a region to bedeleted after using the region in a series of processes, and the like,and related selection information may be generated. Also, the dataprocessing method may be implicitly determined.

For example, a data processing method applied to all the regions (here,the regions TL to BR in Section 7B of FIG. 7 ), which are to be deletedalong with reduction through image resizing, may be one of the method ofsimply removing a region to be deleted, the method of removing a regionto be deleted after using the region in a series of processes, and thelike, and related selection information may be generated. Also, the dataprocessing method may be implicitly determined.

Alternatively, a data processing method applied to each of the regions(here, each of the regions TL to BR in Section 7B of FIG. 7 ), which isto be deleted along with reduction through image resizing, may be one ofthe method of simply removing a region to be deleted, the method ofremoving a region to be deleted after using the region in a series ofprocesses, and the like, and related selection information may begenerated. Also, the data processing method may be implicitlydetermined.

An example in which the resizing is performed according to a resizing(expansion or reduction) operation has been described. In some cases,the description may be applied to an example in which a resizingoperation (here, expansion) is performed and then an inverse resizingoperation (here, reduction) is performed.

For example, a method of filling a region generated along with expansionwith some data of the image may be selected, and then a method ofremoving a region to be deleted along with reduction in the inverseprocess after using the region in a process of restoring or correctingsome data of the image may be selected. Alternatively, a method offilling a region generated along with expansion by copying outer pixelsmay be selected, and then a method of simply removing a region to bedeleted along with reduction in the inverse process may be selected.That is, based on the data processing method selected in the imageresizing process, the data processing method in the inverse process maybe determined.

Unlike the above example, the data processing method of the imageresizing process and the data processing method of the inverse processmay have an independent relationship. That is, irrespective of the dataprocessing method selected in the image resizing process, the dataprocessing method in the inverse process may be selected. For example, amethod of filling a region generated along with expansion by using somedata of the image may be selected, and then a method of simply removinga region to be deleted along with reduction in the inverse process maybe selected.

According to the present invention, the data processing method duringthe image resizing process may be implicitly determined according toencoding/decoding settings, and the data processing method during theinverse process may be implicitly determined according toencoding/decoding settings. Alternatively, the data processing methodduring the image resizing process may be explicitly generated, and thedata processing method during the inverse process may be explicitlygenerated. Alternatively, the data processing method during the imageresizing process may be explicitly generated, and based on the dataprocessing method, the data processing method during the inverse processmay be implicitly determined.

Next, an example in which image resizing is performed in theencoding/decoding apparatus according to an embodiment of the presentinvention will be described. In the following description, as anexample, the resizing process indicates expansion, and the inverseresizing process indicates reduction. Also, a difference between animage before resizing and an image after resizing may refer to an imagesize, and resizing-related information may have some pieces explicitlygenerated and other pieces implicitly determined depending onencoding/decoding settings. Also, the resizing-related information mayinclude information regarding a resizing process and an inverse resizingprocess.

As a first example, a process of resizing an input image may beperformed before encoding is started. The input image may be resizedusing resizing information (e.g., a resizing operation, a resizingdirection, a resizing value, a data processing method, etc.; the dataprocessing method is used during the resizing process) and then may beencoded. The image encoding data (here, an image after resizing) may bestored in the memory after the encoding is complete, and may be added toa bitstream and then transmitted.

A resizing process may be performed before decoding is started. Theimage decoding data may be resized using resizing information (e.g., aresizing operation, a resizing direction, a resizing value, etc.), andthen may be parsed to be decoded. The output image may be stored in thememory after the decoding is complete, and may be changed into the imagebefore resizing by performing the inverse resizing process (here, a dataprocessing method or the like is used; this is used in the inverseresizing process).

As a second example, a process of resizing a reference picture may beperformed before encoding is started. The reference picture may beresized using resizing information (e.g., a resizing operation, aresizing direction, a resizing value, a data processing method, etc.;the data processing method is used during the resizing process) and thenmay be stored in the memory (here, the resized reference picture). Animage may be encoded using the resized reference picture. After theencoding is complete, image encoding data (here, data acquired thoughencoding using the reference picture) may be added to a bitstream andthen transmitted. Also, when the encoded image is stored in the memoryas a reference picture, the above resizing process may be performed.

Before decoding is started, a resizing process for the reference picturemay be performed. The reference picture may be resized using resizinginformation (e.g., a resizing operation, a resizing direction, aresizing value, a data processing method, etc.; the data processingmethod is used during the resizing process) and then may be stored inthe memory (here, the resized reference picture). Image decoding data(here, which is encoded by the encoder using the reference picture) maybe parsed to be decoded. After the decoding is complete, an output imagemay be generated. When the decoded image is stored in the memory as areference picture, the above resizing process may be performed.

As a third example, the resizing process may be performed on an imagebefore filtering of the image (here, a deblocking filter is assumed) andafter encoding (in detail, after encoding, excluding a filteringprocess, is compete). The image may be resized using resizinginformation (e.g., a resizing operation, a resizing direction, aresizing value, a data processing method, etc.; the data processingmethod is used during the resizing), and then the image after resizingmay be generated and then filtered. After the filtering is complete, theinverse resizing process is performed so that the image after resizingmay be changed into the image before resizing.

After decoding is complete (in detail, after decoding, excluding afiltering process, is complete), and before the filtering, the resizingprocess may be performed on the image. The image may be resized usingresizing information (e.g., a resizing operation, a resizing direction,a resizing value, a data processing method, etc.; the data processingmethod is used during the resizing), and then the image after resizingmay be generated and then filtered. After the filtering is complete, theinverse resizing process is performed so that the image after resizingmay be changed into the image before resizing.

In some cases (the first example and the third example), the resizingprocess and the inverse resizing process may be performed. In othercases (the second example), only the resizing process may be performed.

Also, in some cases (the second example and the third example), the sameresizing process may be applied to the encoder and decoder. In othercases (the first example), the same or different resizing processes maybe applied to the encoder and decoder. Here, the resizing processes ofthe encoder and the decoder may differ in terms of the resizingperforming step. For example, in some cases (here, the encoder), theresizing performing step considering image resizing and data processingfor a resized region may be included. In other cases (here, thedecoder), the resizing performing step considering image resizing may beincluded. Here, the former data processing may correspond to the latterdata processing during the inverse resizing process.

Also, in some cases (the third example), the resizing process may beapplied to only a corresponding step, and a resized region may not bestored in the memory. For example, in order to use a resized region in afiltering process, the resized region may be stored in a temporarymemory, filtered, and then removed through the inverse resizing process.In this case, there is no change in size of the image due to resizing.The present invention is not limited to the above example, andmodifications may be made thereto.

The size of the image may be changed through the resizing process, andthus the coordinates of some pixels of the image may be changed throughthe resizing process. This may affect the operation of the picturepartitioning part. According to the present invention, through theprocess, block-based partitioning may be performed on the basis of animage before resizing or an image after resizing. Also, unit (e.g.,tile, slice, etc.)-based partitioning may be performed on the basis ofan image before resizing or an image after resizing, which may bedetermined according to encoding/decoding settings. According to thepresent invention, the following description focuses on a case in whichthe picture partitioning part operates on the basis of the image afterresizing (e.g., the image partitioning process after the resizingprocess), but other modifications may be made. The above example will bedescribed in a plurality of image settings to be described below.

The encoder may add the information generated during the above processto a bitstream in units of at least one of sequences, pictures, slices,tiles, and the like, and the decoder may parse related information fromthe bitstream. Also, the information may be included in the bitstream inthe form of SEI or metadata.

Generally, an input image may be encoded or decoded as it is or afterimage reconstruction. For example, the image reconstruction may beperformed in order to enhance image encoding efficiency, the imagereconstruction may be performed in order to consider network and userenvironments, and the image reconstruction may be performed according tothe type, characteristics, and the like of an image.

According to the present invention, the image reconstruction process mayinclude a reconstruction process solely or in combination with aninverse reconstruction process. The following example description willfocus on the reconstruction process, but the inverse reconstructionprocess may be inversely derived from the reconstruction process.

FIG. 10 is an example diagram of image reconstruction according to anembodiment of the present invention.

It is assumed that Section 10A shows an initial input image. Sections10A to 10D are example diagrams in which an image rotates apredetermined angle including 0 degrees (e.g., a candidate group may begenerated by sampling 360-degrees into k sections; k may have a value of2, 4, 8, or the like; in this example, it is assumed that k is 4).Sections 10E to 10H are example diagrams having an inverse (orsymmetric) relationship with respect to Sections 10A or with respect toSections 10B to 10D.

The start position or scan order of an image may be changed depending onimage reconstruction, but the start position and the scan order may bepredetermined irrespective of the reconstruction, which may bedetermined according to encoding/decoding settings. The followingembodiment assumes that the start position (e.g., an upper left positionof the image) and the scan order (e.g., raster scan) are predeterminedirrespective of image reconstruction.

The image encoding method and the image decoding method according to anembodiment of the present invention may include the following imagereconstruction steps. In this case, the image reconstruction process mayinclude an image reconstruction indication step, an image reconstructiontype identification step, and an image reconstruction performing step.Also, the image encoding apparatus and the image decoding apparatus maybe configured to include an image reconstruction indication part, animage reconstruction type identification part, and an imagereconstruction performing part, which perform the image reconstructionindication step, the image reconstruction type identification step, andthe image reconstruction performing step, respectively. For theencoding, a related syntax element may be generated. For the decoding, arelated syntax element may be parsed.

In the image reconstruction indication step, whether to perform imagereconstruction may be determined. For example, when a signal indicatingimage reconstruction (e.g., convert_enabled_flag) is confirmed, thereconstruction may be performed. When the signal indicating imagereconstruction is not confirmed, the reconstruction may not beperformed, or the reconstruction may be performed by confirming otherencoding/decoding information. Also, although the signal indicatingimage reconstruction is not provided, the signal indicating imagereconstruction may be implicitly activated or deactivated according toencoding/decoding settings (e.g., the characteristics, type, and thelike of an image). When the reconstruction is performed, correspondingreconstruction-related information may be generated or may be implicitlydetermined.

When the signal indicating image reconstruction is provided, thecorresponding signal is a signal for indicating whether to perform theimage reconstruction. Whether to reconstruct a corresponding image maybe determined according to the signal. For example, it is assumed thatthe signal indicating image reconstruction (e.g., convert_enabled_flag)is confirmed. When the corresponding signal is activated (e.g.,convert_enabled_flag=1), the reconstruction may be performed. When thecorresponding signal is deactivated (e.g., convert_enabled_flag=0), thereconstruction may not be performed.

Also, when the signal indicating image reconstruction is not provided,the reconstruction may not be performed, or whether to reconstruct thecorresponding image may be determined by another signal. For example,the reconstruction may be performed according to the characteristics,type, and the like of an image (e.g., a 360-degree image), andreconstruction information may be explicitly generated or may beassigned as a predetermined value. The present invention is not limitedto the above example, and modifications may be made thereto.

In the image reconstruction type identification step, an imagereconstruction type may be identified. The image reconstruction type maybe defined by a reconstruction method, reconstruction mode information,and the like. The reconstruction method (e.g., convert_type_flag) mayinclude flipping, rotation, and the like, and the reconstruction modeinformation may include a mode of the reconstruction method (e.g.,convert_mode). In this case, the reconstruction-related information maybe composed of a reconstruction method and mode information. That is,the reconstruction-related information may be composed of at least onesyntax element. In this case, the number of candidate groups for themode information may be the same or different depending on thereconstruction method.

As an example, the rotation may include candidates having regularintervals (here, 90 degrees) as shown in Sections 10A to 10D. Section10A shows a 0-degree rotation, Section 10B shows a 90-degree rotation,Section 10C shows a 180-degree rotation, and Section 10D shows a270-degree rotation (here, which are measured clockwise).

As an example, the flipping may include candidates as shown in Sections10A, 10E, and 10F. When Section 10A shows no flipping, Sections 10E and10F show a horizontal flipping and a vertical flipping, respectively.

In the above example, settings for rotations having regular intervalsand settings for flippings have been described. However, this is merelyan example of the image reconstruction, and the present invention is notlimited thereto and may include another interval difference, anotherflipping operation, and the like, which may be determined according toencoding/decoding settings.

Alternatively, integrated information (e.g., convert_com_flag) which isgenerated by mixing the reconstruction method and corresponding modeinformation may be included. In this case, the reconstruction-relatedinformation may be mixedly composed of a reconstruction method and modeinformation.

For example, the integrated information may include the candidates asshown in Sections 10A to 10F, which may be examples of a 0-degreerotation, a 90-degree rotation, a 180-degree rotation, a 270-degreerotation, a horizontal flipping, and a vertical flipping with respect toSection 10A.

Alternatively, the integrated information may include the candidates asshown in Sections 10A to 10H, which may be examples of a 0-degreerotation, a 90-degree rotation, a 180-degree rotation, a 270-degreerotation, a horizontal flipping, a vertical flipping, a 90-degreerotation and then horizontal flipping (or a horizontal flipping and then90-degree rotation), and a 90-degree rotation and then vertical flipping(or a vertical flipping and then 90-degree rotation) or examples of a0-degree rotation, a 90-degree rotation, a 180-degree rotation, a270-degree rotation, a horizontal flipping, a 180-degree rotation andthen horizontal flipping (or a horizontal flipping and then 180-degreerotation), a 90-degree rotation and then horizontal flipping (or ahorizontal flipping and then 90-degree rotation), and a 270-degreerotation and then horizontal flipping (or a horizontal flipping and then270-degree rotation).

The candidate group may be configured to include a rotation mode, aflipping mode, and a combination mode of rotation and flipping. Thecombination mode may simply include mode information in thereconstruction method and may include a mode generated by mixing modeinformation in each method. In this case, the combination mode mayinclude a mode generated by mixing at least one mode of some methods(e.g., rotation) and at least one mode of other methods (e.g.,flipping). In the above example, the combination mode includes a casegenerated by combining one mode of some methods with a plurality ofmodes of some methods (here, a 90-degree rotation+multipleflippings/horizontal flipping+multiple rotations). The mixedlyconstructed information may include a case in which reconstruction isnot applied (here, Section 10A) as a candidate group, and the case inwhich reconstruction is not applied may be included as a first candidategroup (e.g., #0 is assigned as an index).

Alternatively, the mixedly constructed information may include modeinformation corresponding to a predetermined reconstruction method. Inthis case, the reconstruction-related information may be composed ofmode information corresponding to a predetermined reconstruction method.That is, information regarding the reconstruction method may be omitted,and the reconstruction-related information may be composed of one syntaxelement associated with the mode information.

For example, the reconstruction-related information may be configured toinclude rotation-specific candidates as shown in Sections 10A to 10D.Alternatively, the reconstruction-related information may be configuredto include flipping-specific candidates as shown in Sections 10A, 10E,and 10F.

An image before the image reconstruction process and an image after theimage reconstruction process may have the same size or at least onedifferent length, which may be determined according to encoding/decodingsettings. The image reconstruction process may be a process ofrearranging pixels in an image (here, an inverse pixel rearrangementprocess is performed during an inverse image reconstruction process;this can be inversely derived from the pixel rearrangement process), andthus the location of at least one pixel may be changed. The pixelrearrangement may be performed according to a rule based on the imagereconstruction type information.

In this case, the pixel rearrangement process may be affected by thesize and shape (e.g., square or rectangle) of an image. In detail, thewidth and height of an image before the reconstruction process and thewidth and height of an image after the reconstruction process may act asvariables during the pixel rearrangement process.

For example, ration information regarding at least one of a ratio of thewidth of the image before the reconstruction process to the width of theimage after the reconstruction process, a ratio of the width of theimage before the reconstruction process to the height of the image afterthe reconstruction process, a ratio of the height of the image beforethe reconstruction process to the width of the image after thereconstruction process, and a ratio of the height of the image beforethe reconstruction process to the height of the image after thereconstruction process (e.g., the former/the latter or the latter/theformer) may act as variables during the pixel rearrangement process.

In the example, when the image before the reconstruction process and theimage after the reconstruction process have the same size, a ratio ofthe width of the image to the height of the image may act as a variableduring the pixel rearrangement process. Also, when the image is in theshape of a square, a ratio of the length of the image before thereconstruction process to the length of the image after thereconstruction process may act as a variable during the pixelrearrangement process.

In the image reconstruction performing step, image reconstruction may beperformed on the basis of identified reconstruction information. Thatis, the image reconstruction may be performed on the basis ofinformation regarding a reconstruction type, a reconstruction mode, andthe like, and encoding/decoding may be performed on the basis of theacquired reconstructed image.

Next, an example in which image reconstruction is performed in theencoding/decoding apparatus according to an embodiment of the presentinvention will be described.

A process of reconstructing an input image may be performed beforeencoding is started. The reconstruction may be performed usingreconstruction information (e.g., an image reconstruction type, areconstruction mode, etc.), and the reconstructed image may be encoded.The image encoding data may be stored in the memory after the encodingis complete, and may be added to a bitstream and then transmitted.

A reconstruction process may be performed before decoding is started.The reconstruction may be performed using reconstruction information(e.g., an image reconstruction type, a reconstruction mode, etc.), andthe image decoding data may be parsed to be decoded. The image may bestored in the memory after the decoding is complete and may be changedto the image before the reconstruction by performing an inversereconstruction process and then output.

The encoder may add the information generated during the above processto a bitstream in units of at least one of sequences, pictures, slices,tiles, and the like, and the decoder may parse related information fromthe bitstream. Also, the information may be included in the bitstream inthe form of SEI or metadata.

TABLE 1 Tile information( ) {  tiles_enabled_flag if(tiles_enabled_flag)  {   num_tile_columns   num_tile_rows  uniform_spacing_flag   if(!uniform_spacing_flag)   {    for(i=0;i<num_tile_columns; i++)     columns_width_tile[i]    for(i=0;i<num_tile_rows; i++)     rows_height_tile[i]   }  tile_header_enabled_flag   if(tile_header_enabled_flag)   {   for(i=0; i< num_tile_columns × num_tiles_rows; i++)    {    tile_coded_flag[i]     if(!tile_coded_flag[i])     {     tile_header( )     }    }   }  } }

Table 1 represents example syntax elements associated with partitioningamong image settings. The following description will focus on anadditional syntax element. Also, in the following example, a syntaxelement is not limited to any specific unit and may be supported invarious units such as a sequence, a picture, a slice, and a tile.Alternatively, the syntax element may be included in SEI, metadata, andthe like. Also, the type, order, condition, and the like of thesupported syntax element in the following example are limited to onlythe example and thus may be changed and determined according toencoding/decoding settings.

In Table 1, tile_header_enabled_flag denotes a syntax element indicatingwhether to support encoding/decoding settings for a tile. When thesyntax element is activated (tile_header_enabled_flag=1),encoding/decoding settings in a tile unit may be provided. When thesyntax element is deactivated (tile_header_enabled_flag=0), theencoding/decoding settings in a tile unit cannot be provided, andencoding/decoding settings in an upper unit may be assigned.

Also, tile_coded_flag denotes a syntax element indicating whether toencode or decode a tile. When the syntax element is activated(tile_coded_flag=1), a corresponding tile may be encoded or decoded.When the syntax element is deactivated (tile_coded_flag=0), thecorresponding tile cannot be encoded or decoded. Here, encoding notbeing performed may mean that encoding data is not generated for acorresponding tile (here, it is assumed that a corresponding region isprocessed by a predetermined rule and the like; applicable to ameaningless region in some projection formats of a 360-degree image).Decoding not being performed means that the decoding data in thecorresponding tile is no longer parsed (here, it is assumed that thecorresponding region is processed by a predetermined rule). Also,decoding data being no longer parsed may mean that encoding data is notpresent in a corresponding unit and thus parsing is no longer performedand may also mean that even through encoding data is present, parsing isno longer performed by the flag. Header information of a tile unit maybe supported according to whether to encode or decode a tile.

The above example focused on a tile. However, the present invention isnot limited to the tile, and the above description may be modified andthen applied to other partitioning units of the present invention. Also,an example of the tile partitioning settings is not limited to the abovecase, and modifications may be made thereto.

TABLE 2 Converting information {  convert_enabled_flag if(convert_enabled_flag)   convert_type_flag }

Table 2 represents example syntax elements associated withreconstruction among image settings.

Referring to Table 2, convert_enabled_flag denotes a syntax elementindicating whether to perform reconstruction. When the syntax element isactivated (convert_enabled_flag=1), a reconstructed image is encoded ordecoded, and additional reconstruction-related information may bechecked. When the syntax element is deactivated(convert_enabled_flag=0), an original image is encoded or decoded.

Also, convert_type_flag denotes mixed information regarding areconstruction method and mode information. One method may be determinedfrom a plurality of candidate groups for a rotation-applied method, aflipping-applied method, and a rotation-and-flipping-applied method.

TABLE 3 Resizing information( ) {  pic_width_in_samples pic_height_in_samples  img_resizing_enabled_flag if(img_resizing_enabled_flag)  {   resizing_met_flag  resizing_mov_flag   if(!resizing_met_flag)   {    width_scale   height_scale   }   else   {    top_height_offset   bottom_height_offset    left_width_offset    right_width_offset   }  resizing_type_flag  } }

Table 3 represents example syntax elements associated with resizingamong image settings.

Referring to Table 3, pic_width_in_samples and pic_height_in_samplesdenote syntax elements indicating the width and the height of an image.The size of an image may be checked through the syntax elements.

Also, img_resizing_enabled_flag denotes a syntax element indicatingwhether to perform image resizing. When the syntax element is activated(img_resizing_enabled_flag=1), an image is encoded or decoded afterresizing, and additional resizing-related information may be checked.When the syntax element is deactivated (img_resizing_enabled_flag=0), anoriginal image is encoded or decoded. Also, the syntax element mayindicate resizing for intra-prediction.

Also, resizing_met_flag indicates a resizing method. One resizing methodmay be determined from a candidate group such as a scale factor-basedresizing method (resizing_met_flag=0), an offset factor-based resizingmethod (resizing_met_flag=1), and the like.

Also, resizing_mov_flag denotes a syntax element for a resizingoperation. For example, one of expansion and reduction may bedetermined.

Also, width_scale and height_scale denote scale factors associated withhorizontal resizing and vertical resizing of the scale factor-basedresizing.

Also, top_height_offset and bottom_height_offset denote an offset factorfor an “up” direction and an offset factor for a “down” direction, whichare associated with horizontal resizing of the offset factor-basedresizing, and left_width_offset and right_width_offset denote an offsetfactor for a “left” direction and an offset factor for a “right”direction, which are associated with vertical resizing of the offsetfactor-based resizing.

The size of an image after resizing may be updated through theresizing-related information and image size information.

Also, resizing_type_flag denotes a syntax element indicating a dataprocessing method for a resized region. The number of candidate groupsfor the data processing method may be the same or different depending onthe resizing method and the resizing operation.

The image setting processes applied to the above-described imageencoding/decoding apparatus may be performed individually or incombination. The following example description will focus on an examplein which the plurality of image setting processes are performed incombination.

FIG. 11 is an example diagram showing images before and after an imagesetting process according to an embodiment of the present invention. Indetail, Section 11A shows an example before image reconstruction isperformed on a partitioned image (e.g., an image projected during360-degree image encoding, and Section 11B shows an image after imagereconstruction is performed on a partitioned image (e.g., an imagepacked during 360-degree image encoding. That is, it can be understoodthat Section 11A is an example diagram before an image setting processis performed and Section 11B is an example diagram after an imagesetting process is performed.

In this example, image partitioning (here, a tile is assumed) and imagereconstruction will be described as the image setting process.

In the following example, the image reconstruction is performed afterthe image partitioning is performed. However, according toencoding/decoding settings, the image partitioning may be performedafter the image reconstruction is performed, and modifications may bemade thereto. Also, the above-described image reconstruction process(including the inverse process) may be applied identically or similarlyto the reconstruction process in the partitioning units in the image inthis embodiment.

The image reconstruction may or may not be performed in all partitioningunits in the image and may be performed in some partitioning units.Accordingly, a partitioning unit before reconstruction (e.g., some of P0to P5) may or may not be the same as a partitioning unit afterreconstruction (e.g., some of S0 to S5). Through the following example,various image reconstruction cases will be described. Also, forconvenience of description, it is assumed that the unit of an image is apicture, the unit of a partitioned image is a tile, and a partitioningunit is in the shape of a rectangle.

As an example, whether to perform image reconstruction may be determinedin some units (e.g., sps_convert_enabled_flag or SEI or metadata, etc.).Alternatively, whether to perform image reconstruction may be determinedin some units (e.g., pps_convert_enabled_flag). This may be allowed whenoccurring in a corresponding unit (here, a picture) for the first timeor when being activated in an upper unit (e.g.,sps_convert_enabled_flag=1). Alternatively, whether to perform imagereconstruction may be determined in some units (e.g.,tile_convert_flag[i]; i is a partitioning unit index). This may beallowed when occurring in a corresponding unit (here, a tile) for thefirst time or when being activated in an upper unit (e.g.,pps_convert_enabled_flag=1). Also, partially, whether to perform imagereconstruction may be implicitly determined according toencoding/decoding settings, and thus related information may be omitted.

As an example, whether to reconstruct partitioning units in an image maybe determined according to a signal indicating image reconstruction(e.g., pps_convert_enabled_flag). In detail, whether to reconstruct allof the partitioning units in the image may be determined according tothe signal. In this case, a single signal indicating imagereconstruction may be generated in the image.

As an example, whether to reconstruct partitioning units in an image maybe determined according to a signal indicating image reconstruction(e.g., tile_convert_flag[i]). In detail, whether to reconstruct some ofthe partitioning units in the image may be determined according to thesignal. In this case, at least one signal indicating imagereconstruction (e.g., a number of signals equal to the number ofpartitioning units) may be generated.

As an example, whether to reconstruct an image may be determinedaccording to a signal indicating image reconstruction (e.g.,pps_convert_enabled_flag[i]), and whether to reconstruct partitioningunits in an image may be determined according to a signal indicatingimage reconstruction (e.g., tile_convert_flag[i]). In detail, when anysignal is activated (e.g., pps_convert_enabled_flag=1), any other signal(e.g., tile_convert_flag[i]) may be additionally checked, and whether toreconstruct some of the partitioning units in the image may bedetermined according to the signal (here, tile_convert_flag[i]). In thiscase, a plurality of signals indicating image reconstruction may begenerated.

When the signal indicating image reconstruction is activated, imagereconstruction-related information may be generated. In the followingexample, a variety of image reconstruction-related information will bedescribed.

As an example, reconstruction information applied to an image may begenerated. In detail, one piece of reconstruction information may beused as reconstruction information for all the partitioning units in theimage.

As an example, reconstruction information applied to partitioning unitsin an image may be generated. In detail, at least one piece ofreconstruction information may be used as reconstruction information forsome of the partitioning units in the image. That is, one piece ofreconstruction information may be used as reconstruction information forone partitioning unit or one pierce of reconstruction information may beused as reconstruction information for a plurality of partitioningunits.

The following example will be described in combination with an examplein which image reconstruction is performed.

For example, when the signal indicating image reconstruction (e.g.,pps_convert_enabled_flag) is activated, reconstruction informationapplied in common to partitioning units in an image may be generated.Alternatively, when the signal indicating image reconstruction (e.g.,pps_convert_enabled_flag) is activated, reconstruction informationapplied individually to the partitioning units in the image may begenerated. Alternatively, when the signal indicating imagereconstruction (e.g., tile_convert_flag[i]) is activated, reconstructioninformation applied individually to the partitioning units in the imagemay be generated. Alternatively, when the signal indicating imagereconstruction (e.g., tile_convert_flag[i]) is activated, reconstructioninformation applied in common to the partitioning units in the image maybe generated.

The reconstruction information may be implicitly or explicitly processeddepending on encoding/decoding settings. For the implicit processing,the reconstruction information may be assigned as a predetermined valuedepending on the characteristics, type, and the like of the image.

P0 to P5 in Section 11A may correspond to S0 to S5 in Section 11B, andthe reconstruction process may be performed on partitioning units. Forexample, P0 may not be reconstructed and then may be assigned to S0. P1may be rotated by 90 degrees and then may be assigned to S1. P2 may berotated by 180 degrees and then may be assigned to S2. P3 may behorizontally flipped and then may be assigned to S3. P4 may be rotatedby 90 degrees and horizontally flipped and then may be assigned to S4.P5 may be rotated by 180 degrees and horizontally flipped and then maybe assigned to S5.

However, the present invention is not limited to the above example, andvarious modifications may be made thereto. Like the above example, thepartitioning units in the image may not be reconstructed, or at leastone of reconstruction using rotation, reconstruction using flipping, andreconstruction using rotation and flipping in combination may beperformed.

When image reconstruction is applied to partitioning units, anadditional reconstruction process such as partitioning unitrearrangement may be performed. That is, the image reconstructionprocess according to the present invention may be configured to includerearrangement of partitioning units in an image as well as rearrangementof pixels in an image and may be represented using some syntax elementsin Table 4 (e.g., part_top, part_left, part_width, part_height, and thelike). This means that the image partitioning process and the imagereconstruction process may be understood in combination. In the aboveexample, it has been described that an image is partitioned into aplurality of units.

P0 to P5 in Section 11A may correspond to S0 to S5 in Section 11B, andthe reconstruction process may be performed on partitioning units. Forexample, P0 may not be reconstructed and then may be assigned to S0. P1may not be reconstructed and then may be assigned to S2. P2 may berotated by 90 degrees and then may be assigned to S1. P3 may behorizontally flipped and then may be assigned to S4. P4 may be rotatedby 90 degrees and horizontally flipped and then may be assigned to S5.P5 may be horizontally flipped and then rotated by 180 degrees and thenmay be assigned to S3. The present invention is not limited thereto, andalso various modifications may be made thereto.

Also, P_Width and P_Height of FIG. 7 may correspond to P_Width andP_Height of FIG. 11 , and P′_Width and P′_Height of FIG. 7 maycorrespond to P′_Width and P′_Height of FIG. 11 . The size of the imageafter resizing in FIG. 7 , which is P′_Width×P′_Height, may be expressedas (P_Width+Exp_L+Exp_R)×(P_Height+Exp_T+Exp_B), and the size of theimage after resizing in FIG. 11 , which is P′_Width×P′_Height, may beexpressed as(P_Width+Var0_L+Var1_L+Var2_L+Var0_R+Var1_R+Var2_R)×(P_Height+Var0_T+Var1_T+Var0_B+Var1_B)or(Sub_P0_Width+Sub_P1_Width+Sub_P2_Width+Var0_L+Var1_L+Var2_L+Var0_R+Var1_R+Var2_R)×(Sub_P0_Height+Sub_P1_Height+Var0_T+Var1_T+Var0_B+Var1_B).

Like the above example, for the image reconstruction, rearrangement ofpixels in partitioning units of an image may be performed, rearrangementof partitioning units in an image may be performed, and both of therearrangement of pixels in partitioning units of an image and therearrangement of partitioning unit in an image may be performed. In thiscase, the rearrangement of partitioning units in an image may beperformed after the rearrangement of pixels in partitioning units isperformed, or the rearrangement of pixels in partitioning units may beperformed after the rearrangement of partitioning units in an image isperformed.

Whether to perform the rearrangement of partitioning units in an imagemay be determined according to a signal indicating image reconstruction.Alternatively, a signal for the rearrangement of the partitioning unitsin the image may be generated. In detail, when a signal indicating imagereconstruction is activated, the signal may be generated. Alternatively,the signal may be implicitly or explicitly processed depending onencoding/decoding settings. For the implicit processing, the signal maybe determined depending on the characteristics, type, and the like ofthe image.

Also, information regarding the rearrangement of partitioning units inan image may be implicitly or explicitly performed depending onencoding/decoding settings and may be determined according to thecharacteristics, type, and the like of the image. That is, each of thepartitioning units may be arranged according to arrangement informationpredetermined for the partitioning units.

Next, an example in which partitioning units in an image arereconstructed in the encoding/decoding apparatus according to anembodiment of the present invention will be described.

A partitioning process may be performed on an input image usingpartitioning information before encoding is started. A reconstructionprocess may be performed on partitioning units using reconstructioninformation, and an image reconstructed for each partitioning unit maybe encoded. The image encoding data may be stored in the memory afterthe encoding is complete, and may be added to a bitstream and thentransmitted.

A partitioning process may be performed using partitioning informationbefore decoding is started. A reconstruction process may be performed onpartitioning units using reconstruction information, and the imagedecoding data may be parsed to be decoded in the reconstructedpartitioning units. The image decoding data may be stored in the memoryafter the decoding is complete, and a plurality of partitioning unitsare merged into a single unit after an inverse reconstruction process inthe partitioning units is performed, and thus an image may be output.

FIG. 12 is an example diagram of resizing each partitioning unit of animage according to an embodiment of the present invention. P0 to P5 ofFIG. 12 correspond to P0 to P5 of FIG. 11 , and S0 to S5 of FIG. 12correspond to S0 to S5 of FIG. 11 .

In the following example, the description will focus on a case in whichimage resizing is performed after image partitioning is performed.However, image partitioning may be performed after image resizing isperformed, depending on encoding/decoding settings, and modificationsmay be made thereto. Also, the above-described image resizing process(including the inverse process) may be applied identically or similarlyto the image partitioning unit resizing process in this embodiment.

For example, TL to BR of FIG. 7 may correspond to TL to BR ofpartitioning units SX (S0 to S5) of FIG. 12 . S0 and S1 of FIG. 7 maycorrespond to PX and SX of FIG. 12 . P_Width and P_Height of FIG. 7 maycorrespond to Sub_PX_Width and Sub_PX_Height of FIG. 12 . P′_Width andP′_Height of FIG. 7 may correspond to Sub_SX_Width and Sub_SX_Height ofFIG. 12 . Exp_L, Exp_R, Exp_T, and Exp_B of FIG. 7 may correspond toVarX_L, VarX_R, VarX_T, and VarX_B of FIG. 12 , and other factors maycorrespond.

The process of resizing partitioning units in the image in sections 12Ato 12F may differ from image expansion or reduction in Sections 7A and7B of FIG. 7 in that settings for the image expansion or reduction maybe present in proportion to the number of partitioning units. Also, theprocess of resizing partitioning units in the image may differ from theimage expansion or reduction in terms of having settings applied incommon or individually to the partitioning units in the image. In thefollowing example, various resizing cases will be described, and theresizing process may be performed in consideration of theabove-description.

According to the present invention, the image resizing may or may not beperformed on all partitioning units in the image and may be performed onsome partitioning units. Through the following example, various imageresizing cases will be described. Also, for convenience of description,it is assumed that the resizing operation is for expansion, the resizingoperation is based on an offset factor, the resizing direction is an“up” direction, a “down” direction, a “left” direction, and a “right”direction, the resizing direction is set to operate by the resizinginformation, the unit of an image is a picture, and the unit of apartitioned image is a tile.

As an example, whether to perform image resizing may be determined insome units (e.g., sps_img_resizing_enabled_flag or SEI or metadata,etc.). Alternatively, whether to perform image resizing may bedetermined in some units (e.g., pps_img_resizing_enabled_flag). This maybe allowed when occurring in a corresponding unit (here, a picture) forthe first time or when being activated in an upper unit (e.g.,sps_img_resizing_enabled_flag=1). Alternatively, whether to performimage resizing may be determined in some units (e.g.,tile_resizing_flag[i]; i is a partitioning unit index). This may beallowed when occurring in a corresponding unit (here, a tile) for thefirst time or when being activated in an upper unit. Also, partially,whether to perform image resizing may be implicitly determined accordingto encoding/decoding settings, and thus related information may beomitted.

As an example, whether to resize partitioning units in an image may bedetermined according to a signal indicating image resizing (e.g.,pps_img_resizing_enabled_flag). In detail, whether to resize allpartitioning units in an image may be determined according to thesignal. In this case, a single signal indicating image resizing may begenerated.

As an example, whether to resize partitioning units in an image may bedetermined according to a signal indicating image resizing (e.g.,tile_resizing_flag[i]). In detail, whether to resize some partitioningunits in an image may be determined according to the signal. In thiscase, at least one signal indicating image resizing (e.g., a number ofsignals equal to the number of partitioning units) may be generated.

As an example, whether to resize an image may be determined according toa signal indicating image resizing (e.g.,pps_img_resizing_enabled_flag), and whether to resize partitioning unitsin an image may be determined according to a signal indicating imageresizing (e.g., tile_resizing_flag[i]). In detail, when any signal isactivated (e.g., pps_img_resizing_enabled_flag=1), any other signal(e.g., tile_resizing_flag[i]) may be additionally checked, and whetherto resizing some partitioning units in an image may be performedaccording to the signal (here, tile_resizing_flag[i]). In this case, aplurality of signals indicating image resizing may be generated.

When the signal indicating image resizing is activated, imageresizing-related information may be generated. In the following example,a variety of image resizing-related information will be described.

As an example, resizing information applied to an image may begenerated. In detail, one piece of resizing information or a set ofpieces of resizing information may be used as resizing information forall partitioning units in an image. For example, one piece of resizinginformation applied in common to an “up” direction, a “down” direction,a “left” direction, and a “right” direction of partitioning units in animage (or a resizing value applied to all the resizing directionssupported or allowed in partitioning units; in this example, one pieceof information) or a set of pieces of resizing information appliedindividually to the “up” direction, the “down” direction, the “left”direction, and the “right” direction (or a number of pieces of resizinginformation equal to the number of resizing directions allowed orsupported by the partitioning unit; in this example, up to four piecesof information) may be generated.

As an example, resizing information applied to partitioning units in animage may be generated. In detail, at least one piece of resizinginformation or a set of pieces of resizing information may be used asresizing information for all partitioning units in an image. That is,one piece of resizing information or a set of pieces of resizinginformation may be used as resizing information for one partitioningunit or as resizing information for a plurality of partitioning units.For example, a piece of resizing information applied in common to an“up” direction, a “down” direction, a “left” direction, and a “right”direction of one partitioning unit in an image may be generated, or aset of pieces of resizing information individually applied to the “up”direction, the “down” direction, the “left” direction, and the “right”direction may be generated. Alternatively, a piece of resizinginformation applied in common to an “up” direction, a “down” direction,a “left” direction, and a “right” direction of a plurality ofpartitioning units in an image may be generated, or a set of pieces ofresizing information individually applied to the “up” direction, the“down” direction, the “left” direction, and the “right” direction may begenerated. The configuration of the resizing set means resizing valueinformation regarding at least one resizing direction.

In summary, resizing information applied in common to partitioning unitsin an image may be generated. Alternatively, resizing informationapplied individually to partitioning units in an image may be generated.The following example will be described in combination with an examplein which image resizing is performed.

For example, when the signal indicating image resizing (e.g.,pps_img_resizing_enabled_flag) is activated, resizing informationapplied in common to partitioning units in an image may be generated.Alternatively, when the signal indicating image resizing (e.g.,pps_img_resizing_enabled_flag) is activated, resizing informationapplied individually to partitioning units in an image may be generated.Alternatively, when the signal indicating image resizing (e.g.,tile_resizing_flag[i]) is activated, resizing information appliedindividually to partitioning units in an image may be generated.Alternatively, when the signal indicating image resizing (e.g.,tile_resizing_flag[i]) is activated, resizing information applied incommon to partitioning units in an image may be generated.

The resizing direction, the resizing information, and the like of theimage may be implicitly or explicitly processed depending onencoding/decoding settings. For the implicit processing, the resizinginformation may be assigned as a predetermined value depending on thecharacteristics, type, and the like of the image.

It has been described that the resizing direction in the resizingprocess of the present invention may be at least one of the “up”direction, the “down” direction, the “left” direction, and the “right”direction and the resizing direction and the resizing information may beprocessed explicitly or implicitly. That is, a resizing value (including0; this means no resizing) may be implicitly predetermined for somedirections, and a resizing value (including 0; this means no resizing)may be explicitly assigned for other directions.

In even a partitioning unit in an image, the resizing direction and theresizing information may be set to be implicitly or explicitlyprocessed, and this may be applied to the partitioning unit in theimage. For example, a setting applied to one partitioning unit in animage may occur (here, a number of settings equal to the number ofpartitioning units may occur), a setting applied to a plurality ofpartitioning units in an image may occur, or a setting applied to allpartitioning units in an image may occur (here, one setting may occur),and at last one setting may occur in an image (e.g., one to a number ofsettings equal to the number of partitioning units may occur). Thesetting information applied to partitioning units in an image may becollected, and a single set of settings may be defined.

FIG. 13 is an example diagram of a set of resizing or setting of apartitioning unit in an image.

In detail, FIG. 13 illustrates various examples of implicitly orexplicitly processing a resizing direction and resizing information forpartitioning units in an image. In the following example, forconvenience of description, the implicit processing assumes thatresizing values of some resizing directions are 0.

As shown in Section 13A, the resizing may be explicitly processed whenthe boundary of a partitioning unit matches the boundary of an image(here, a thick solid line), and the resizing may be implicitly processedwhen the boundary of a partitioning unit does not match the boundary ofan image (a thin solid line). For example, P0 may be resized in an “up”direction and a “left” direction (a2 and a0), P1 may be resized in an“up” direction (a2), P2 may be resized in an “up” direction and a“right” direction (a2 and a1), P3 may be resized in a “down” directionand a “left” direction (a3 and a0), P4 may be resized in a “down”direction (a3), and P5 may be resized in a “down” direction and a“right” direction (a3 and a1). In this case, the resizing may not beallowed in the other directions.

As shown in Section 13B, some directions (here, up and down) of apartitioning unit may allow the resizing to be explicitly processed, andsome directions (here, left and right) of a partitioning unit may allowthe resizing to be explicitly processed (here, a thick solid line) whenthe boundary of the partitioning unit matches the boundary of the imageand may allow the resizing to be implicitly processed (here, a thinsolid line) when the boundary of the partitioning unit does not matchthe boundary of the image. For example, P0 may be resized in an “up”direction, a “down” direction, and a “left” direction (b2, b3, and b0),P1 may be resized in an “up” direction and a “down” direction (b2 andb3), P2 may be resized in an “up” direction, a “down” direction, and a“right” direction (b2, b3, and b1), P3 may be resized in an “up”direction, a “down” direction, and a “left” direction (b3, b4, and b0),P4 may be resized in an “up” direction and a “down” direction (b3 andb4), and P5 may be resized in an “up” direction, a “down” direction, anda “right” direction (b3, b4, and b1). In this case, the resizing may notbe allowed in the other directions.

As shown in Section 13C, some directions (here, left and right) of apartitioning unit may allow the resizing to be explicitly processed, andsome directions (here, up and down) of a partitioning unit may allow theresizing to be explicitly processed (here, a thick solid line) when theboundary of the partitioning unit matches the boundary of the image andmay allow the resizing to be implicitly processed (here, a thin solidline) when the boundary of the partitioning unit does not match theboundary of the image. For example, P0 may be resized in an “up”direction, a “left” direction, and a “right” direction (c4, c0, and c1),P1 may be resized in an “up” direction, a “left” direction, and a“right” direction (c4, c1, and c2), P2 may be resized in an “up”direction, a “left” direction, and a “right” direction (c4, c2, and c3),P3 may be resized in a “down” direction, a “left” direction, and a“right” direction (c5, c0, and c1), P4 may be resized in a “down”direction, a “left” direction, and a “right” direction (c5, c1, and c2),and P5 may be resized in a “down” direction, a “left” direction, and a“right” direction (c5, c2, and c3). In this case, the resizing may notbe allowed in the other directions.

Settings related to image resizing like the above example may havevarious cases. A plurality of sets of settings are supported so thatsetting set selection information may be explicitly generated, or apredetermined setting set may be implicitly determined according toencoding/decoding settings (e.g., the characteristics, type, and thelike of the image).

FIG. 14 is an example diagram in which both of a process of resizing animage and a process of resizing partitioning units in an image arerepresented.

Referring to FIG. 14 , the process of resizing an image and the inverseprocess may proceed in directions e and f, and the process of resizingpartitioning units in an image and the inverse process may proceed indirections d and g. That is, a resizing process may be performed on animage, and then a resizing process may be performed on partitioningunits in an image. The resizing order may not be fixed. This means thata plurality of resizing processes may be possible.

In summary, the image resizing process may be classified into resizingof an image (or resizing an image before partitioning) and resizing ofpartitioning units in an image (or resizing an image afterpartitioning). Neither, either, or both of the resizing of an image andthe resizing of partitioning units in an image may be performed, whichmay be determined according to encoding/decoding settings (e.g., thecharacteristics, type, and the like of the image).

When in the example, a plurality of resizing processes are performed,the resizing of an image may be performed in at least one of the “up”direction, the “down” direction, the “left” direction, and the “right”direction of the image, and at least one of the partitioning units inthe image may be resized. In this case, the resizing may be performed inat least one of an “up” direction, a “down” direction, a “left”direction, and a “right” direction of the partitioning unit to beresized.

Referring to FIG. 14 , the size of an image before resizing (A) may bedefined as P_Width×P_Height, the size of an image after primary resizing(or an image before secondary resizing; B) may be defined asP′_Width×P′_Height, and the size of an image after secondary resizing(or an image after final resizing; C) may be defined asP″_Width×P″_Height. The image before resizing (A) denotes an image onwhich no resizing is performed, the image after primary resizing (B)denotes an image on which some resizing is performed, and the imageafter secondary resizing (C) denotes an image on which all resizing isperformed. For example, the image after primary resizing (B) may denotean image in which resizing is performed in partitioning units of theimage as shown in Sections 13A to 13C, and the image after secondaryresizing (C) may denote an image acquired by entirely resizing the imageafter primary resizing (B) as shown in Section 7A of FIG. 7 . Theopposite case is also possible. However, the present invention is notlimited to the above example, and various modifications may be madethereto.

In the size of the image after primary resizing (B), P′_Width may beacquired through P_Width and at least one horizontal resizing value thatis laterally resized, and P′_Height may be acquired through P_Height andat least one vertical resizing value that is longitudinally resized. Inthis case, the resizing value may be a resizing value generated inpartitioning units.

In the size of the image after secondary resizing (C), P″_Width may beacquired through P′_Width and at least one horizontal resizing valuethat is laterally resized, and P″_Height may be acquired throughP′_Height and at least one vertical resizing value that islongitudinally resized. In this case, the resizing value may be aresizing value generated in the image.

In summary, the size of the image after resizing may be acquired throughat least one resizing value and the size of the image before resizing.

In a resized region of the image, information regarding a dataprocessing method may be generated. Through the following example,various data processing methods will be described. A data processingmethod generated during the inverse resizing process may be appliedidentically or similarly to that of the resizing process. The dataprocessing methods in the resizing process and the inverse resizingprocess will be described through various combinations to be describedbelow.

As an example, a data processing method applied to an image may begenerated. In detail, one data processing method or a set of dataprocessing methods may be used as a data processing method for allpartitioning units in the image (here, it is assumed that all thepartitioning units are to be resized). For example, one data processingmethod applied in common to an “up” direction, a “down” direction, a“left” direction, and a “right” direction of a partitioning unit in animage (or a data processing method or the like applied to all resizingdirections supported or allowed in partitioning units; in this example,one piece of information) or a set of data processing methods applied tothe “up” direction, the “down” direction, the “left” direction, and the“right” direction (or a number of data processing methods equal to thenumber of resizing directions supported or allowed in partitioningunits; in this example, up to four pieces of information) may begenerated.

As an example, a data processing method applied to partitioning units inan image may be generated. In detail, at least one data processingmethod or a set of data processing methods may be used as a dataprocessing method for some partitioning units in the image (here, it isassumed that the partitioning units are to be resized). That is, onedata processing method or a set of data processing methods may be usedas a data processing method for one partitioning unit or a dataprocessing method for a plurality of partitioning units. For example,one data processing method applied in common to an “up” direction, a“down” direction, a “left” direction, and a “right” direction of onepartitioning unit in an image may be generated, or a set of dataprocessing methods individually applied to the “up” direction, the“down” direction, the “left” direction, and the “right” direction may begenerated. Alternatively, one data processing method applied in commonto an “up” direction, a “down” direction, a “left” direction, and a“right” direction of a plurality of partitioning units in an image maybe generated, or a set of data processing methods individually appliedto the “up” direction, the “down” direction, the “left” direction, andthe “right” direction may be generated. The configuration of the set ofdata processing methods means a data processing method for at least oneresizing direction.

In summary, a data processing method applied in common to partitioningunits in an image may be used. Alternatively, a data processing methodapplied individually to partitioning units in an image may be used. Thedata processing method may use a predetermined method. The predetermineddata processing method may be provided as at least one method. Thiscorresponds to an implicit process, and selection information for thedata processing method may be explicitly generated, which may bedetermined according to encoding/decoding settings (e.g., thecharacteristics, type, and the like of the image).

That is, a data processing method applied in common to partitioningunits in an image may be used. A predetermined method may be used, orone of a plurality of data processing methods may be selected.Alternatively, a data processing method applied individually topartitioning units in an image may be used. Depending on thepartitioning units, a predetermined method may be used, or one of aplurality of data processing methods may be selected.

In the following example, some cases in which partitioning units in animage are resized (here, it is assumed that the resizing is forexpansion) will be described (here, a resized region is filled with somedata of the image).

Specific regions TL to BR of some units (e.g., S0 to S5 in Sections 12Ato 12F) may be resized using data of specific regions tl to br of someunits P0 to P5 (in Sections 12A to 12F). In this case, the some unitsmay be the same as (e.g., S0 and P0) or different from (e.g., S0 and P1)one another. That is, the regions TL to BR to be resized may be filledwith some data tl to br of a corresponding partitioning unit and may befilled with some data of a partitioning unit other than thecorresponding partitioning unit.

As an example, the regions TL to BR of which the current partitioningunit is resized may be resized using data tl to br of the currentpartitioning unit. For example, TL of S0 may be filled with data tl ofP0, RC of S1 may be filled with data tr+rc+br of P1, BL+BC of S2 may befilled with data bl+bc+br of P2, and TL+LC+BL of S3 may be filled withdata tl+lc+bl of P3.

As an example, the regions TL to BR of which the current partitioningunit is resized may be resized using data tl to br of a partitioningunit that is spatially adjacent to the current partitioning unit. Forexample, TL+TC+TR of S4 may be filled with data bl+bc+br of P1 in the“up” direction, BL+BC of S2 may be filled with data tl+tc+tr of P5 inthe “down” direction, LC+BL of S2 may be filled with data tl+rc+bl of P1in the “left” direction, RC of S3 may be filled with data tl+lc+bl of P4in the “right” direction, and BR of S0 may be filled with data tl of P4in a “down+left” direction.

As an example, the regions TL to BR of which the current partitioningunit is resized may be resized using data tl to br of a partitioningunit that is not spatially adjacent to the current partitioning unit.For example, data in a (e.g., horizontal, vertical, etc.) boundaryregion between both ends of an image may be acquired. LC of S3 may beacquired using data tr+rc+br of S5, RC of S2 may be acquired using datatl+lc of S0, BC of S4 may be acquired using data tc+tr of S1, and TC ofS1 may be acquired using data bc of S4.

Alternatively, data of specific regions (a region that is not spatiallyadjacent to but determined to have high correlation with a resizedregion) of the image may be acquired. BC of S1 may be acquired usingdata tl+lc+bl of S3, RC of S3 may be acquired using data tl+tc of S1,and RC of S5 may be acquired using data bc of S0.

Also, some cases in which a partitioning unit in an image is resized(here, it is assumed that the resizing is for reduction) are as follows(here, removal is performed through restoration or correction using somedata of an image).

Specific regions TL to BR of some units (e.g., S0 to S5 in Sections 12Ato 12F) may be used in a restoration or correction process for specificregions tl to br of some units P0 to P5. In this case, the some unitsmay be the same as (e.g., S0 and P0) or different from (e.g., S0 and P2)one another. That is, the region to be resized may be used to restoresome data of a corresponding partitioning unit and then removed, and theregion to be resized may be used to restore some data of a partitioningunit other than the corresponding partitioning unit and then removed.The detailed example may be inversely derived from the expansionprocess, and thus will be omitted.

The example may be applied to a case in which data with high correlationis present in a region to be resized, and information regardinglocations referenced for the resizing may be explicitly generated orimplicitly acquired according to a predetermined rule. Alternatively,related information may be checked in combination. This may be anexample that may be applied when data is acquired from another regionwith continuity in encoding of a 360-degree image.

Next, an example in which partitioning units in an image are resized inthe encoding/decoding apparatus according to an embodiment of thepresent invention will be described.

A partitioning process may be performed on an input image beforeencoding is started. A resizing process may be performed on apartitioning unit using resizing information, and the image may beencoded after the partitioning unit is resized. The image encoding datamay be stored in the memory after the encoding is complete, and may beadded to a bitstream and then transmitted.

A partitioning process may be performed using partitioning informationbefore decoding is started. A resizing process may be performed onpartitioning units using resizing information, and the image decodingdata may be parsed to be decoded in the resized partitioning units. Theimage decoding data may be stored in the memory after the decoding iscomplete, and the plurality of partitioning units are merged into asingle unit after an inverse resizing process for the partitioning unitsis performed, and thus an image may be output.

Another example of the above-described image resizing process may beapplied. The present invention is not limited thereto, and modificationsmay be made thereto.

In the image setting process, the image resizing and the imagereconstruction may be allowed to be combined. The image reconstructionmay be performed after the image resizing is performed. Alternatively,the image resizing may be performed after the image reconstruction isperformed. Also, the image partitioning, the image reconstruction, andthe image resizing may be allowed to be combined. The image resizing andthe image reconstruction may be performed after the image partitioningis performed. The order of image settings is not fixed and may bechanged, which may be determined according to encoding/decodingsettings. In this example, the image setting process will be describedas the image reconstruction and the image resizing being performed afterthe image partitioning is performed. However, depending onencoding/decoding settings, another order is possible, and alsomodifications may be made thereto.

For example, the image setting process may be performed in the followingorder: partitioning→reconstruction; reconstruction→partitioning;partitioning→resizing; resizing->partitioning; resizing→reconstruction;reconstruction→resizing; partitioning->reconstruction→resizing;partitioning→resizing→reconstruction;resizing→partitioning->reconstruction;resizing→reconstruction→partitioning;reconstruction→partitioning->resizing; andreconstruction→resizing→partitioning, and a combination with additionalimage settings may be possible. As described above, the image settingprocess may be sequentially performed, but some or all of the settingprocess may be simultaneously performed. Also, as some of the imagesetting process, a plurality of processes may be performed according toencoding/decoding settings (e.g., the characteristics, type, and thelike of an image). The following example indicates various combinationsof the image setting process.

As an example, P0 to P5 in Section 11A may correspond to S0 to S5 inSection 11 B, and the reconstruction process (here, rearrangement ofpixels) and the resizing process (here, resizing of partitioning unitsto have the same size) may be performed in partitioning units. Forexample, P0 to P5 may be resized based on offset and may be assigned toS0 to S5. Also, P0 may not be reconstructed and then may be assigned toS0. P1 may be rotated by 90 degrees and then may be assigned to S1. P2may be rotated by 180 degrees and then may be assigned to S2. P3 may berotated by 270 degrees and then may be assigned to S3. P4 may behorizontally flipped and then may be assigned to S4. P5 may bevertically flipped and then may be assigned to S5.

As an example, P0 to P5 in Section 11A may correspond to positions thatare the same as or different from S0 to S5 in Section 11 B, and thereconstruction process (here, rearrangement of pixels and partitioningunits) and the resizing process (here, resizing of partitioning units tohave the same size) may be performed in partitioning units. For example,P0 to P5 may be resized based on scale and may be assigned to S0 to S5.Also, P0 may not be reconstructed and then may be assigned to S0. P1 maynot be reconstructed and then may be assigned to S2. P2 may be rotatedby 90 degrees and then may be assigned to S1. P3 may be horizontallyflipped and then may be assigned to S4. P4 may be rotated by 90 degreesand horizontally flipped and then may be assigned to S5. P5 may behorizontally flipped and then rotated by 180 degrees and then may beassigned to S3.

As an example, P0 to P5 in Section 11A may correspond to E0 to E5 inSection 5E, and the reconstruction process (here, rearrangement ofpixels and partitioning units) and the resizing process (here, resizingof partitioning units to have the different sizes) may be performed inpartitioning units. For example, P0 may not be resized and reconstructedand then may be assigned to E0, P1 may be resized based on scale but isnot reconstructed and then may be assigned to E1, P2 may not be resizedbut reconstructed and then may be assigned to E2, P3 may be resizedbased on offset but is not reconstructed and then may be assigned to E4,P4 may not be resized but reconstructed and may be assigned to E5, andP5 may be resized based on offset and reconstructed and then may beassigned to E3.

Like the above example, the absolute position or the relative positionof the partitioning units before and after the image setting process inthe image may be maintained or changed, which may be determinedaccording to encoding/decoding settings (e.g., the characteristics,type, and the like of the image). Also, various combinations of theimage setting processes may be possible. The present invention is notlimited thereto, and thus various modifications may be made thereto.

The encoder may add the information generated during the above processto a bitstream in units of at least one of sequences, pictures, slices,tiles, and the like, and the decoder may parse related information fromthe bitstream. Also, the information may be included in the bitstream inthe form of SEI or metadata.

TABLE 4 Partition information( ) {  parts_enabled_flag if(parts_enabled_flag)  {   num_partitons   for(i=0; i<num_partitions;i++)   {    part_top[i]    part_left[i]    part_width[i]   part_height[i]   }   part_header_enabled_flag  if(part_header_enabled_flag)    partition_header( )  } }

Table 4 represents example syntax elements associated with a pluralityof image settings. The following description will focus on an additionalsyntax element. Also, in the following example, a syntax element is notlimited to any specific unit and may be supported in various units suchas a sequence, a picture, a slice, and a tile. Alternatively, the syntaxelement may be included in SEI, metadata, and the like.

Referring to Table 4, parts_enabled_flag denotes a syntax elementindicating whether to partition some units. When the syntax element isactivated (parts_enabled_flag=1), an image may be partitioned into aplurality of units, and the plurality of units may be encoded ordecoded. Also, additional partitioning information may be checked. Whenthe syntax element is deactivated (parts_enabled_flag=0), an originalimage is encoded or decoded. In this example, the description will focuson a rectangular partitioning unit such as a tile, and differentsettings for the existing tile and partitioning information may beprovided.

Here, num_partitions refers to a syntax element indicating the number ofpartitioning units, and num_partitions plus 1 is equal to the number ofpartitioning units.

Also, part_top[i] and part_left[i] refer to syntax elements indicatinglocation information of the partitioning units and denote horizontalstart positions and vertical start positions of the partitioning units(e.g., upper left positions of the partitioning units). Also,part_width[i] and part_height[i] refer to syntax elements indicatingsize information of the partitioning units and denote the widths and theheights of the partitioning units. In this case, the start positions andthe size information may be set in pixel units or in block units. Also,the syntax element may be a syntax element that may be generated duringthe image reconstruction process or a syntax element that may begenerated when the image partitioning process and the imagereconstruction process are constructed in combination.

Also, part_header_enabled_flag denotes a syntax element indicatingwhether to support encoding/decoding settings for a partitioning unit.When the syntax element is activated (part_header_enabled_flag=1),encoding/decoding settings for a partitioning unit may be provided. Whenthe syntax element is deactivated (part_header_enabled_flag=0), theencoding/decoding settings cannot be provided, and encoding/decodingsettings for an upper unit may be assigned.

The above example is not limited to an example of syntax elementsassociated with resizing and reconstruction in a partitioning unit amongimage settings, and modifications may be made thereto as otherpartitioning units and settings of the present invention. This examplehas been described under the assumption that the resizing and thereconstruction are performed after the partitioning is performed, butthe present invention is not limited thereto, and modifications may bemade thereto in another image setting order or the like. Also, the type,order, condition, and the like of the supported syntax element in thefollowing example are limited to only the example and thus may bechanged and determined according to encoding/decoding settings.

TABLE 5 Converting information {  convert_enabled_flag if(convert_enabled_flag)  {   for(i=0; i<num_partitions; i++)   {   part_convert_flag[i]    if(part_convert_flag[i])    convert_type_flag[i]   }  } }

Table 5 represents example syntax elements associated withreconstruction in a partitioning unit among image settings.

Referring to Table 5, part_convert_flag[i] denotes a syntax elementindicating whether to reconstruct a partitioning unit. The syntaxelement may be generated for each partitioning unit. When the syntaxelement is activated (part_convert_flag[i]=1), the reconstructedpartitioning unit may be encoded or decoded, and additionalreconstruction-related information may be checked. When the syntaxelement is deactivated (part_convert_flag[i]=0), an originalpartitioning unit is encoded or decoded. Here, convert_type_flag[i]refers to mode information regarding reconstruction of a partitioningunit and may be information regarding pixel rearrangement.

Also, a syntax element indicating additional reconstruction such aspartitioning unit rearrangement may be generated. In this example, thepartitioning unit rearrangement may be performed through part_top andpart_left, which are syntax element indicating the above imagepartitioning, or a syntax element (e.g., index information) associatedwith the partitioning unit rearrangement may be generated.

TABLE 6 Resizing information {  img_resizing_enabled_flag if(img_resizing_enabled_flag)  {   resizing_met_flag  resizing_mov_flag   for(i=0; i<num_partitions; i++)   {   part_resizing_flag[i]    if(part_resizing_flag[i])    {    if(!resizing_met_flag)     {      width_scale[i]     height_scale[i]     }     else     {      top_height_offset[i]     bottom_height_offset[i]      left_width_offset[i]     right_width_offset[i]     }     for(j=0; j<num_offset; j++)     resizing_type_flag[i][j]    }   }  } }

Table 6 represents example syntax elements associated with resizing in apartitioning unit among image settings.

Referring to Table 6, part_resizing_flag[i] denotes a syntax elementindicating whether to resize a partitioning unit in an image. The syntaxelement may be generated for each partitioning unit. When the syntaxelement is activated (part_resizing_flag[i]=1), the resized partitioningunit may be encoded or decoded after resizing, and additionalresizing-related information may be checked. When the syntax element isdeactivated (part_resizing_flag[i]=0), an original partitioning unit isencoded or decoded.

Also, width_scale[i] and height_scale[i] denote scale factors associatedwith horizontal resizing and vertical resizing of the scale factor-basedresizing in a partitioning unit.

Also, top_height_offset[i] and bottom_height_offset[i] denote an offsetfactor for an “up” direction and an offset factor for a “down”direction, which are associated with the offset factor-based resizing ina partitioning unit, and left_width_offset[i] and right_width_offset[i]denote an offset factor for a “left” direction and an offset factor fora “right” direction, which are associated with the offset factor-basedresizing in a partitioning unit.

Also, resizing_type_flag[i][j] denotes a syntax element indicating adata processing method for a resized region in a partitioning unit. Thesyntax element denotes an individual data processing method for aresizing direction. For example, a syntax element indicating anindividual data processing method for a resized region in an “up”direction, a “down” direction, a “left” direction, and a “right”direction may be generated. The syntax element may be generated on thebasis of resizing information (e.g., which may be generated only whenresizing is performed in some directions).

The above-described image setting process may be a process that isapplied according to the characteristics, type, and the like of theimage. In the following example, the above-described image settingprocess may be applied without or with any change, even without specialmention. In the following example, the description will focus on a caseof an addition to or a change in the above example.

For example, a 360-degree image or an omnidirectional image generatedthrough a 360-degree camera has different characteristics from those ofan image acquired through a general camera and has a different encodingenvironment from that of compression of a normal image.

Unlike a normal image, a 360-degree image may have no boundary part withdiscontinuity, and data of all regions of the 360-degree image may havecontinuity. Also, an apparatus such as an HMD may require ahigh-definition image because an image should be replayed in front ofeyes through a lens. When an image is acquired through a stereoscopiccamera, the amount of image data processed may increase. Various imagesetting processes considering a 360-degree image may be performed toprovide efficient encoding environments including the above example.

The 360-degree camera may be a plurality of cameras or a camera having aplurality of lenses and sensors. The camera or lens may cover alldirections around any center point captured by the camera.

The 360-degree image may be encoded using various methods. For example,the 360-degree image may be encoded using various image processingalgorithms in a 3D space, and may be converted into a 2D space andencoded using various image processing algorithms. According to thepresent invention, the following description will focus on a method ofconverting a 360-degree image into a 2D space and encoding or decodingthe converted image.

A 360-degree image encoding apparatus according to an embodiment of thepresent invention may include some or all of the elements shown in FIG.1 , and may further include a pre-processing unit configured topre-process an input image (Stitching, Projection, Region-wise Packing).Meanwhile, a 360-degree image decoding apparatus according to anembodiment of the present invention may include some or all of theelements shown in FIG. 2 , and may further include a post-processingunit configured to post-process an encoded image before decoding theencoded image to reproduce an output image.

In other words, the encoder may pre-process an input image, encode thepre-processed image, and transmit a bitstream including the image, andthe decoder may parse, decode, and post-process the transmittedbitstream to generate an output image. In this case, the transmittedbitstream may include information generated during the pre-processingprocess and information generated during the encoding process, and thebitstream may be parsed and used during the decoding process and thepost-processing process.

Subsequently, an operation method for a 360-degree image encoder will bedescribed in more detail, and an operation method for a 360-degree imagedecoder may be easily derived by those skilled in the art because theoperation method for the 360-degree image decoder is opposite to theoperation method for the 360-degree image encoder, and thus a detaileddescription thereof will be omitted.

The input image may be subject to performing a stitching and projectionprocess on a sphere-based 3D projection structure, and image data on the3D projection structure may be projected into a 2D image through theprocess.

The projected image may be configured to include some or all of360-degree content according to encoding settings. In this case,location information of a region (or a pixel) to be placed at the centerof the projected image may be implicitly generated as a predeterminedvalue or may be explicitly generated. Also, when the projected imageincludes specific regions of the 360-degree content, the rangeinformation and location information of the included regions may begenerated. Also, range information (e.g., the width and the height) andlocation information (e.g., which is measured on the basis of an upperleft end of an image) of a region of interest (ROI) may be generatedfrom the projected image. In this case, a specific region with highimportance in the 360-degree content may be set as an ROI. The360-degree image may allow all content in an “up” direction, a “down”direction, a “left” direction, and a “right” direction to be viewed, buta user's gaze may be limited to a portion of the image, which may be setas an ROI in consideration of the limitation. For the purpose ofefficient encoding, an ROI may be set to have good quality and highresolution, and the other regions may be set to have lower quality andlower resolution than the ROI.

Among a plurality of 360-degree image transmission schemes, a singlestream transmission scheme may allow a full image or a viewport image tobe transmitted in an individual single bitstream for a user. Amulti-stream transmission scheme may allow several full images withdifferent image qualities to be transmitted in multiple bitstreams, andthus an image quality may be selected according to user environments andcommunication conditions. A tiled-stream transmission scheme may allow atile unit-based partial image that is individually encoded to betransmitted in multiple bitstreams, and thus a tile may be selectedaccording to user environments and communication conditions.Accordingly, the 360-degree image encoder may generate and transmit abitstream having two or more qualities, and the 360-degree image decodermay set an ROI according to a user's view and may selectively decode thebitstream according to the ROI. That is, a place where a user's gaze isdirected may be set as an ROI through a head tracking or eye trackingsystem, and only the necessary part may be rendered.

The projected image may be converted into a packed image obtained byperforming a region-wise packing process. The region-wise packingprocess may include a step of partitioning a projected image into aplurality of regions, and the partitioned regions may be arranged (orrearranged) in the image packed according to the region-wise packingsettings. The region-wise packing may be performed to increase spatialcontinuity when a 360-degree image is converted into a 2D image (or aprojected image). Thus, it is possible to reduce the size of the imagethrough the region-wise packing. Also, the region-wise packing may beperformed to reduce deterioration in image quality caused duringrendering, enable a viewport-based projection, and provide other typesof projection formats. The region-wise packing may or may not beperformed depending on encoding settings, which may be determined on thebasis of a signal indicating whether to perform the region-wise packing(e.g., regionwise_packing_flag; only when regionwise_packing_flag isactivated, information regarding the region-wise packing may begenerated).

When the region-wise packing is performed, setting information (ormapping information) in which specific regions of the projected imageare assigned (or arranged) to specific regions of the packed image maybe displayed (or generated). When the region-wise packing is notperformed, the projected image and the packed image may be the sameimage.

In the above-description, a stitching process, a projection process, anda region-wise packing process are defined as individual processes, butsome (e.g., stitching+projection, projection+ region-wise packing) orall (e.g., stitching+projection+ region-wise packing) of the processesmay be defined as a single process.

At least one packed image may be generated from the same input imageaccording to settings for the stitching process, the projection process,and the region-wise packing process. Also, according to the settings forthe region-wise packing process, at least one piece of encoding data forthe same projected image may be generated.

The packed image may be partitioned by performing a tiling process. Inthis case, the tiling, which is a process in which an image ispartitioned into a plurality of regions and then transmitted, may be anexample of the 360-degree image transmission schemes. As describedabove, the tiling may be performed for the purpose of partial decodingin consideration of user environments and may also be performed for thepurpose of efficient processing of vast data of 360-degree images. Forexample, when an image is composed of one unit, the entire image may bedecoded to decode an ROI. On the other hand, when an image is composedof a plurality of unit regions, it may be efficient to decode only anROI. In this case, the partitioning may be performed in tile units,which are partitioning units according to a conventional encodingscheme, or may be performed in various partitioning units (e.g., aquadrilateral partitioning block, etc.) that have been describedaccording to the present invention. Also, the partitioning unit may be aunit for performing independent encoding/decoding. The tiling may beperformed independently or on the basis of the projected image or thepacked image. That is, the partitioning may be performed on the basis ofa face boundary of the projected image, a face boundary of the packedimage, packing settings, etc., and may be independently performed foreach partitioning unit. This may affect generation of partitioninginformation during the tiling process.

Next, the projected image or the packed image may be encoded. Encodingdata and information generated during the pre-processing process may beadded to a bitstream, and the bitstream may be transmitted to the360-degree image decoder. The information generated during thepre-processing process may be added to the bitstream in the form of SEIor metadata. In this case, the bitstream may contain at least one pieceof encoding data having partially different settings for the encodingprocess and at least one piece of pre-processing information havingpartially different settings for the pre-processing process. This is toconstruct a decoded image in combination of a plurality of pieces ofencoding data (encoding data+pre-processing information) according touser environments. In detail, the decoded image may be constructed byselectively combining the plurality of pieces of encoding data. Also,the process may be performed while being separated into two parts toapply to a binocular system, and the process may be performed on anadditional depth image.

FIG. 15 is an example diagram showing a 2D planar space and a 3D spaceshowing a 3D image.

Generally, for the purpose of a 360-degree 3D virtual space, threedegrees of freedom (3DoF) may be needed, and three rotations may besupported with respect to an X axis (Pitch), a Y axis (Yaw), and a Zaxis (Roll). The DoF refers to the degree of freedom in space, 3DoFrefers to the degree of freedom that includes rotations around the Xaxis, Y axis, and Z axis, as shown in Section 15A, and 6DoF refers tothe degree of freedom that additionally allows movements along the Xaxis, Y axis, and Z axis as well as 3DoF. The following description willfocus on the image encoding apparatus and the image decoding apparatusof the present invention having 3DoF. When 3DoF or greater (3DoF+) issupported, the image encoding apparatus and the image decoding apparatusmay be modified or combined with an additional process or apparatus thatis not shown.

Referring to Section 15A, Yaw may have a range from −π (−180 degrees) toπ (180 degrees), Pitch may have a range from −π/2 rad (or −90 degrees)to π/2 rad (or 90 degrees), and Roll may have a range from −π/2 rad (or−90 degrees) to π/2 rad (or 90 degrees). In this case, when it isassumed that ϕ and θ are longitude and latitude in the maprepresentation of the earth, 3D space coordinates (x, y, z) may betransformed from 2D space coordinates (ϕ, θ). For example, 3D spacecoordinates may be derived from 2D space coordinates according totransformation formulas x=cos(θ) cos(ϕ), y=sin(θ), and z=−cos(θ) sin(ϕ).

Also, (ϕ, θ) may be transformed into (x, y, z). For example, 2D spacecoordinates may be derived from 3D space coordinates according totransformation formulas ϕ=tan−1(−Z/X) and θ=sin−1(Y/(X2+Y2+Z2)1/2).

When a pixel in the 3D space is accurately transformed into a pixel inthe 2D space (e.g., an integer unit pixel in the 2D space), the pixel inthe 3D space may be mapped to the pixel in the 2D space. When a pixel inthe 3D space is not accurately transformed into a pixel in the 2D space(e.g., a decimal unit pixel in the 2D space), a pixel acquired throughinterpolation may be mapped to the 2D pixel. In this case, as theinterpolation, nearest neighbor interpolation, bi-linear interpolation,B-spline interpolation, bi-cubic interpolation, or the like may be used.In this case, related information may be explicitly generated byselecting one of the plurality of interpolation candidates, or aninterpolation method may be implicitly determined according to apredetermined rule. For example, a predetermined interpolation filtermay be used according to a 3D model, a projection format, a colorformat, and a slice/tile type. Also, when the interpolation informationis explicitly generated, information regarding filter information (e.g.,a filter coefficient) may be included.

Section 15B shows an image in which a 3D space is transformed into a 2Dspace (a 2D planar coordinate system). (ϕ,θ) may be sampled (i,j) on thebasis of the size (the width and height) of an image. Here, i may have arange from 0 to P_Width−1, and j may have a range from 0 to P_Height−1.

(ϕ,θ) may be a center point (or a reference point; a point depicted as Cof FIG. 15 ; coordinates (ϕ,θ)=(0,0)) for arranging a 360-degree imagewith respect to the projected image. The setting for the center pointmay be designated in the 3D space, and location information for thecenter point may be explicitly generated or implicitly determined as apredetermined value. For example, center position information in Yaw,center position information in Pitch, center position information inRoll, and the like may be generated. When a value for the information isnot specified separately, each value may be assumed to be zero.

An example in which the entire 360-degree image is transformed from the3D space into the 2D space has been described above, but specificregions of the 360-degree image may be transformed, and locationinformation (e.g., some locations belonging to the region; in thisexample, location information regarding the center point), rangeinformation, and the like for the specific regions may be explicitlygenerated or may implicitly follow predetermined location and rangeinformation. For example, center position information in Yaw, centerposition information in Pitch, center position information in Roll,range information in Yaw, range information in Pitch, range informationin Roll, and the like may be generated, and specific regions may be atleast one region. Thus, location information, range information, and thelike of a plurality of regions may be processed. When a value for theinformation is not specified separately, the entire 360-degree image maybe assumed.

H0 to H6 and W0 to W5 in Section 15A indicate some latitudes andlongitudes in Section 15B, which may be expressed as coordinates (C, j)and (i, C) (C is a longitude or latitude component) in Section 15B.Unlike a general image, when a 360-degree image is converted into the 2Dspace, distortions may occur or warpage of content in an image mayoccur. This may depend on the region of the image, and differentencoding/decoding settings may be applied to the location of the imageor regions partitioned according to the location. When theencoding/decoding settings are adaptively applied on the basis ofencoding/decoding information in the present invention, the locationinformation (e.g., an x component, a y component, or a range defined byx and y) may be included as an example of the encoding/decodinginformation.

The description of the 3D space and the 2D space is defined to assistthe description of the embodiments of the present invention. However,the present invention is not limited thereto, and the above descriptionmay be modified in terms of details or may be applied to other cases.

As described above, an image acquired through a 360-degree camera may betransformed into a 2D space. In this case, a 360-degree image may bemapped using a 3D model, and various 3D models such as a sphere, a cube,a cylinder, a pyramid, and a polyhedron may be used. When the 360-degreeimage mapped based on the model is transformed into the 2D space, aprojection process may be performed according to a projection formatbased on the model.

FIGS. 16A to 16D are conceptual diagrams illustrating a projectionformat according to an embodiment of the present invention.

FIG. 16A illustrates an Equi-Rectangular Projection (ERP) format inwhich a 360-degree image is projected into a 2D plane. FIG. 16Billustrates a CubeMap Projection (CMP) format in which a 360-degreeimage is projected to a cube. FIG. 16C illustrates an OctaHedronProjection (OHP) format in which a 360-degree image is projected to anoctahedron. FIG. 16D illustrates an IcoSahedral Projection (ISP) formatin which a 360-degree image is projected to a polyhedron. However, thepresent invention is not limited thereto, and various projection formatsmay be used. In FIGS. 16A to 16D, the left sides show 3D modes, and theright sides show examples of transformation into the 2D space throughthe projection process. Various size and shapes may be providedaccording to the projection format. Each shape may be composed ofsurfaces or faces, and each face may be expressed as a circle, atriangle, a quadrangle, etc.

In the present invention, the projection format may be defined by a 3Dmode, face settings (e.g., the number of faces, the shape of faces, theshape configuration of faces, etc.), projection process settings, etc.When at least one element is different in the definition, the projectionformat may be regarded as a different projection format. For example,the ERP is composed of a sphere model (the 3D model), one face (thenumber of faces), and a quadrangular face (the shape of faces). However,when some (e.g., a formula used during transformation from the 3D spaceinto the 2D space; that is, an element that has the same remainingprojection settings and makes a difference in at least one pixel of aprojected image in the projection process) of the settings for theprojection process are different, the format may be classified as adifferent format such as ERP1 and ERP2. As another example, the CMP iscomposed of a cube model, six faces, and a quadrangular face. When some(e.g., a sampling method applied during transformation from the 3D spaceinto the 2D space) of the settings during the projection process aredifferent, the format may be classified as a different format such asCMP1 and CMP2.

When a plurality of projection formats are used instead of onepredetermined projection format, projection format identificationinformation (or projection format information) may be explicitlygenerated. The projection format identification information may beconfigured through various methods.

As an example, a projection format may be identified by assigning indexinformation (e.g., proj_format_flag) to a plurality of projectionformats. For example, #0 may be assigned to ERP, #1 may be assigned toCMP, #2 may be assigned to OHP, #3 may be assigned to ISP, #4 may beassigned to ERP1, #5 may be assigned to CMP1, #6 may be assigned toOHP1, #7 may be assigned to ISP1, #8 may be assigned to CMP compact, #9may be assigned to OHP compact, #10 may be assigned to ISP compact, and#11 or higher may be assigned to other formats.

As an example, the projection format may be identified using at leastone piece of element information constituting the projection format. Inthis case, as the element information constituting the projectionformat, 3D model information (e.g., 3d_model_flag; #0 indicates asphere, #1 indicates a cube, #2 indicates a cylinder, #3 indicates apyramid, #4 indicates polyhedron 1, and #5 indicates polyhedron 2), facenumber information (e.g., num_face_flag; a method of increasing by 1,starting from 1; the number of faces generated in the projection formatis assigned as index information, that is, #0 indicates one, #1indicates three, #2 indicates six, #3 indicates eight, and #4 indicatestwenty), face shape information (e.g., shape_face_flag; #0 indicates aquadrangle, #1 indicates a circle, #2 indicates a triangle, #3 indicatesa quadrangle+a circle, and #4 indicates a quadrangle+a triangle),projection process setting information (e.g., 3d_2d_convert_idx), andthe like may be included.

As an example, the projection format may be identified using projectionformat index information and element information constituting theprojection format. For example, as the projection format indexinformation, #0 may be assigned to ERP, #1 may be assigned to CMP, #2may be assigned to OHP, #3 may be assigned to ISP, and #4 or greater maybe assigned to other formats. The projection format (e.g., ERP, ERP1,CMP, CMP1, OHP, OHP1, ISP, and ISP1) may be identified along with theelement information constituting the projection format (here, theprojection process setting information). Alternatively, the projectionformat (e.g., ERP, CMP, CMP compact, OHP, OHP compact, ISP, and ISPcompact) may be identified along with the element informationconstituting the projection format (here, region-wise packing).

In summary, the projection format may be identified using the projectionformat index information, may be identified using at least one piece ofthe projection format element information, and may be identified usingthe projection format index information and at least one of theprojection format element information. This may be defined according toencoding/decoding settings. In the present invention, the followingdescription assumes that the projection format is identified using theprojection format index. In this example, the description will focus ona projection format that is expressed using faces with the same size andshape, but a configuration having different faces in size and shape maybe possible. Also, the configuration of each face may be the same as ordifferent from those shown in FIGS. 16A to 16D, the number of each faceis used as a symbol for identifying a corresponding face, and there isno limitation on a specific order. For convenience of description, thefollowing description assumes that, with respect to the projected image,ERP is a projection format including one face+a quadrangle, CMP is aprojection format including six faces+a quadrangle, OHP is a projectionformat including eight faces+a triangle, ISP is a projection formatincluding twenty faces+a triangle, and the faces have the same size andshape. However, the description may be identically or similarly appliedeven to different settings.

As shown in FIGS. 16A to 16D, the projection format may be classified asone face (e.g., ERP) or a plurality of faces (e.g., CMP, OHP, and ISP).Also, the shape of each face may be classified as a quadrangle, atriangle, or the like. The classification may be an example of the type,characteristics, and the like of the image according to the presentinvention, which may be applied when different encoding/decodingsettings are provided depending on the projection format. For example,the type of an image may be a 360-degree image, and the characteristicsof an image may be one of the classifications (e.g., each projectionformat, a projection format having one face or a plurality of faces, aprojection format having a quadrangular face or a non-quadrangularface).

A 2D planar coordinate system (e.g., (l, j)) may be defined in each faceof a 2D projected image, and the characteristics of the coordinatesystem may differ depending on the projection format, the location ofeach face, and the like. ERP may have one 2D planar coordinate system,and other projection formats may have a plurality of 2D planarcoordinate systems depending on the number of faces. In this case, thecoordinate system may be expressed as (k,i,j), and k may indicate indexinformation for each face.

FIG. 17 is a conceptual diagram showing that a projection format isincluded in a rectangular image according to an embodiment of thepresent invention.

That is, it may be understood that Sections 17A to 17C show that theprojection formats of FIGS. 16B to 16D are implemented as a rectangularimage.

Referring to Sections 17A to 17C, each image format may be configured ina rectangular shape to encode or decode a 360-degree image. For ERP, asingle coordinate system may be used as it is. However, for otherprojection formats, coordinate systems of faces may be integrated into asingle coordinate system, and a detailed description thereof will beomitted.

Referring to Sections 17A to 17C, while a rectangular image isconstructed, it may be confirmed that a region filled with meaninglessdata such as a void or background is generated. That is, the rectangularimage may be composed of a region including actual data (here, a face;an active area) and a meaningless region added to construct therectangular image (here, assuming that the region is filled with anypixel value; an inactive area). This may reduce performance due to anincrease in encoding data, that is, an increase in image size caused bythe meaningless region as well as encoding/decoding of actual imagedata.

Accordingly, a process for constructing an image by excluding themeaningless region and using the region including actual data may beadditionally performed.

FIG. 18 is a conceptual diagram of a method of converting a projectionformat into a rectangular shape, that is, a method of performingrearrangement on a face to exclude a meaningless region according to anembodiment of the present invention.

Referring to Sections 18A to 18C, an example for rearranging Sections17A to 17C may be confirmed, and this process may be defined as aregion-wise packing process (CMP compact, OHP compact, ISP compact, andthe like). In this case, the face may be not only rearranged but alsopartitioned and then rearranged (OHP compact, ISP compact, and thelike). This may be performed in order to remove the meaningless regionas well as to improve encoding performance through efficient facearrangement. For example, when an image is continuously arranged betweenfaces (e.g., B2-B3-B1, B5-60-64, etc. in Section 18A), predictionaccuracy upon encoding is enhanced, and thus encoding performance may beenhanced. Here, the region-wise packing according to the projectionformat is merely an example, and the present invention is not limitedthereto.

FIG. 19 is a conceptual diagram showing that a regional packing processis performed to convert a CMP projection format into a rectangular imageaccording to an embodiment of the present invention.

Referring to Sections 19A to 19C, a CMP projection format may bearranged as 6×1, 3×2, 2×3, and 1×6. Also, when some faces are resized,the arrangement may be made as shown in Sections 19D and 19E. InSections 19A to 19E, CMP is applied as an example. However, the presentinvention is not limited thereto, and other projection formats may beapplied. The arrangement of faces of an image acquired through theregion-wise packing may follow a predetermined rule corresponding to theprojection format or may explicitly generate information regarding thearrangement.

360-degree image encoding and decoding apparatuses according to anembodiment of the present invention may be configured to include some orall of the elements of the image encoding and decoding apparatuses shownin FIGS. 1 and 2 . In particular, a format transformation partconfigured to transform a projection format and an inverse formattransformation part configured to inversely transform a projectionformat may be further included in the image encoding apparatus and theimage decoding apparatus, respectively. That is, an input image may beprocessed through the format transformation part and then encoded by theimage encoding apparatus of FIG. 1 , and a bitstream may be decoded andthen processed through the inverse format transformation part by theimage decoding apparatus of FIG. 2 to generate an output image. Thefollowing description will focus on the process performed by the encoder(here, input image, encoding, etc.), and the process performed by thedecoder may be inversely derived from the encoder. Also, redundantdescription of the aforementioned will be omitted.

The following description assumes that the input image is the same as apacked image or a 2D projected image which is acquired by the 360-degreeencoding apparatus performing the pre-processing process. That is, theinput image may be an image acquired by performing the projectionprocess according to some projection formats or the region-wise packingprocess. A projection formation pre-applied to the input image may beone of the various projection formats, which may be regarded as a commonformat and referred to as a first format.

The format transformation part may perform transformation into aprojection format other than the first format. In this case, theprojection format into which the transformation is to be performed maybe referred to as a second format. For example, ERP may be set as thefirst format and may be transformed into the second format (e.g., ERP2,CMP, OHP, and ISP). In this case, ERP2 has a kind of EPR format havingthe same conditions, such as a 3D model and a face configuration, butsome different settings. Alternatively, projection formats may be thesame format having the same projection format settings (e.g., ERP=ERP2)and may have different image sizes or resolutions. Alternatively, someof the following image setting processes may be applied. For convenienceof description, such an example has been mentioned, but each of thefirst format and the second format may be one of the various projectionformats. However, the present invention is not limited thereto, andmodifications may be made thereto.

During the format transformation process, a pixel of an image aftertransformation (an integer pixel) may be acquired from a decimal unitpixel, as well as an integer unit pixel, in an image beforetransformation due to different coordinate system characteristics, andthus interpolation may be performed. An interpolation filter used inthis case may be the same as or similar to that described above. In thiscase, related information may be explicitly generated by selecting oneof a plurality of interpolation filter candidates, or the interpolationfilter may be implicitly determined according to a predetermined rule.For example, a predetermined interpolation filter may be used accordingto a projection format, a color format, and a slice/tile type. Also,when the interpolation filter is explicitly provided, informationregarding filter information (e.g., a filter coefficient) may beincluded.

In the format transformation part, the projection format may be definedas including region-wise packing, etc. That is, projection andregion-wise packing may be performed during the format transformationprocess. Alternatively, after the format transformation process, aprocess such as region-wise packing may be performed before encoding isperformed.

The encoder may add the information generated during the above processto a bitstream in units of at least one of sequences, pictures, slices,tiles, and the like, and the decoder may parse related information fromthe bitstream. Also, the information may be included in the bitstream inthe form of SEI or metadata.

Next, an image setting process applied to the 360-degree imageencoding/decoding apparatus according to an embodiment of the presentinvention will be described. The image setting process according to thepresent invention may be applied to a pre-processing process, apost-processing process, a format transformation process, an inverseformat transformation process, and the like of the 360-degree imageencoding/decoding apparatus as well as general encoding/decodingprocesses. The following description of the image setting process willfocus on the 360-degree image encoding apparatus and may contain theabove-described image settings. Redundant description of theaforementioned image setting process will be omitted. Also, thefollowing example will focus on the image setting process, and theinverse image setting process may be inversely derived from the imagesetting process. Some cases may be confirmed through the aforementionedvarious embodiment of the present invention.

The image setting process according to the present invention may beperformed in the 360-degree image projection step, the region-wisepacking step, the format transformation step, or other steps.

FIG. 20 is a conceptual diagram of 360-degree image partitioningaccording to an embodiment of the present invention. In FIG. 20 , it isassumed that an image is projected by ERP.

Section 20A illustrates an image projected by ERP, and the image may bepartitioned using various methods. In the example, the descriptionfocuses on a slice or tile, and it is assumed that W0 to W2 and H0 andH1 are partitioning boundary lines for a slice or tile and follow araster scan order. The following example focuses on a slice and a tile.However, the present invention is not limited thereto, and anotherpartitioning method may be applied thereto.

For example, the partitioning may be performed in slice units, and H0and H1 may be provided as partitioning boundaries. Alternatively, thepartitioning may be performed in tile units, and W0 to W2, H0 and H1 maybe provided as partitioning boundaries.

Section 20B illustrates an example in which an image projected by ERP ispartitioned into tiles (it is assumed to have the same tile partitioningboundaries (W0 to W2, H0, and H1 are all activated) as shown in Section20A). When it is assumed that a region P is the entire image and aregion V is a region on which a user's gaze stays or a viewport, theremay be various methods in order to provide an image corresponding to theviewport. For example, the region corresponding to the viewport may beacquired by decoding the entire image (e.g., tiles a to i). In thiscase, the entire image may be decoded, and the tile a to i (here, aregion A+a region B) may be decoded when the image is partitioned.Alternatively, the region corresponding to the viewport may be acquiredby decoding a region belonging to the viewport. In this case, when theimage is partitioned, the region corresponding to the viewport may beacquired from an image restored by decoding tiles f, g, j, and k (here,the region B). The former case may be referred to as full decoding (orviewport independent coding), and the latter case may be referred to aspartial decoding (or viewport dependent coding). The latter case may bean example that may occur in a 360-degree image with a large amount ofdata. The tile unit-based partitioning method may be more frequentlyused than the slice unit-based partitioning method in that a partitionedregion may be flexibly acquired. For the partial decoding,referenceability of a partitioning unit may be spatially or temporarilylimited (here, implicitly processed) because it is not possible to findwhere the viewpoint will occur, and the encoding/decoding may beperformed in consideration the limitation. The following example will bedescribed, focusing on the full decoding, but 360-degree imagepartitioning will be described, focusing on a tile (or a rectangularpartitioning method of the present invention) in order to prepare forthe partial decoding. However, the following description may be appliedto other partitioning units in the same manner or in a modified manner.

FIG. 21 is an example diagram of 360-degree image partitioning and imagereconstruction according to an embodiment of the present invention. InFIG. 21 , it is assumed that an image is projected by CMP.

Section 21A illustrates an image projected by CMP, and the image may bepartitioned using various methods. It is assumed that W0 to W2, H0, andH1 are partitioning boundary lines of a face, a slice, and a tile andfollow a raster scan order.

For example, the partitioning may be performed in slice units, and H0and H1 may be provided as partitioning boundaries. Alternatively, thepartitioning may be performed in tile units, and W0 to W2, H0 and H1 maybe provided as partitioning boundaries. Alternatively, the partitioningmay be performed in face units, and W0 to W2, H0 and H1 may be providedas partitioning boundaries. In this example, it is assumed that the faceis a part of the partitioning unit.

In this case, the face may be a partitioning unit (here, dependentencoding/decoding) which is performed to classify or distinguish regionshaving different properties (here, a plane coordinate system of eachface) in the same image according to the characteristics, type (in theexample, a 360-degree image and an projection format), and the like ofthe image while the slice or tile may be a partitioning unit (here,independent encoding/decoding) which is performed to partition an imageaccording to user definitions. Also, the face may be a unit which ispartitioned by a predetermined definition (or inducement from theprojection format information) during a projection process according tothe projection format while the slice or tile may be a unit which ispartitioned by explicitly generating partitioning information accordingto user definitions. Also, the face may have a polygonal partitioningshape including a quadrangle according to the projection format, theslice may have any partitioning shape that cannot be defined as aquadrangle or a polygon, and the tile may have a quadrangularpartitioning shape. The setting of the partitioning unit may be definedonly for the description of this example.

In the example, it has been described that the face is a partitioningunit classified for a region distinction. However, the face may be aunit for performing independent encoding/decoding according toencoding/decoding settings as at least one face unit, and may havesettings for performing independent encoding/decoding in combinationwith a tile, a slide, and the like. In this case, explicit informationof the tile and the slice may be generated when the face is combinedwith a tile, a slice, and the like, or the tile and the slice may beimplicitly combined on the basis of face information. Alternatively, theexplicit information of the tile and the slice may be generated on thebasis of the face information.

As a first example, one image partitioning process (here, a face) isperformed, and image partitioning may implicitly omit partitioninginformation (which is acquired from projection format information). Thisexample is for dependent encoding/decoding settings and may be anexample corresponding to a case in which referenceability between faceunits is not limited.

As a second example, one image partitioning process (here, a face) isperformed, and image partitioning may explicitly generate partitioninginformation. This example is for dependent encoding/decoding settingsand may be an example corresponding to a case in which referenceabilitybetween face units is not limited.

As a third example, a plurality of image partitioning processes (here, aface and a tile) are performed, some image partitioning (here, a face)may implicitly omit or explicitly generate partitioning information, andother image partitioning (here, a tile) may explicitly generatepartitioning information. In this example, one image partitioningprocess (here, a face) precedes the other image partitioning process(here, a tile).

As a fourth example, a plurality of image partitioning processes areperformed, some image partitioning (here, a face) may implicitly omit orexplicitly generate partitioning information, and other imagepartitioning (here, a tile) may explicitly generate partitioninginformation on the basis of the some image partitioning (here, a face).In this example, one image partitioning process (here, a face) precedesthe other image partitioning process (here, a tile). In some cases (thesecond example is assumed) of this example, it may be the same that thepartitioning information is explicitly generated, but there may be adifference in partitioning information configuration.

As a fifth example, a plurality of image partitioning processes areperformed, some image partitioning (here, a face) may implicitly omitpartitioning information, and other image partitioning (here, a tile)may omit implicitly partitioning information on the basis of the someimage partitioning (here, a face). For example, a face unit may beindividually set as a tile unit, or a plurality of face units (here,when adjacent faces have continuity, the face units are grouped;otherwise, the face units are not grouped; B2-B3-B1 and B4-60-B5 inSection 18A) may be set as a tile unit. According to a predeterminedrule, a face unit may be set as a tile unit. This example is forindependent encoding/decoding settings and may be an examplecorresponding to a case in which referenceability between face units islimited. That is, in some cases (the first example is assumed), it maybe the same that the partitioning information is implicitly processed,but there may be a difference in encoding/decoding settings.

The example may be a description of a case in which the partitioningprocess may be performed in the projection step, the region-wise packingstep, the initial encoding/decoding step, and the like, and may be anyother image partitioning process performed in the encoder/decoder.

In Section 21A, a rectangular image may be constructed by adding aregion B, which does not include data, to a region A, which includesdata. In this case, the location, size, shape, number, and the like ofthe region A and the region B may be information that may be checkedthrough a projection format or the like or information that may bechecked when information regarding a projected image is explicitlygenerated, and related information may be represented with theabove-described image partitioning information, image reconstructioninformation, and the like. For example, information (e.g., part_top,part_left, part_width, part_height, and part_convert_flag) regardingspecific regions of the projected image may be represented as shown inTable 4 and Table 5. However, the present invention is not limitedthereto and may be applied to other cases (e.g., another projectionformat, other projection settings, etc.).

The region B and the region A may be constructed as a single image andthen encoded or decoded. Alternatively, the partitioning may beperformed in consideration of region-wise characteristics, and differentencoding/decoding settings may be applied. For example, encoding ordecoding may not be performed on the region B by using informationregarding whether to perform encoding or decoding (e.g., tile_coded_flagwhen it is assumed that the partitioning unit is a tile). In this case,a corresponding region may be restored to certain data (here, any pixelvalue) according to a predetermined rule. Alternatively, in theabove-described image partitioning process, the region B may havedifferent encoding/decoding settings from the region A. Alternatively, acorresponding region may be removed by performing the region-wisepacking process.

Section 21 B shows an example in which an image packed by CMP ispartitioned into tiles, slices, or faces. In this case, the packed imageis an image on which a face rearrangement process or a region-wisepacking process is performed and may be an image acquired by performingthe image partitioning and image reconstruction according to the presentinvention.

In Section 21 B, a rectangular shape may be constructed to include aregion including data. In this case, the location, size, shape, number,and the like of the region may be information that may be checkedthrough a predetermined setting or information that may be checked wheninformation regarding the packed image is explicitly generated, andrelated information may be represented with the above-described imagepartitioning information, image reconstruction information, and thelike. For example, information (e.g., part_top, part_left, part_width,part_height, and part_convert_flag) regarding a specific region of thepacked image may be represented as shown in Table 4 and Table 5.

The packed image may be partitioned using various partitioning methods.For example, the partitioning may be performed in slice units, and H0may be provided as a partitioning boundary. Alternatively, thepartitioning may be performed in tile units, and W0, W1, and H0 may beprovided as partitioning boundaries. Alternatively, the partitioning maybe performed in face units, and W0, W1, and H0 may be provided aspartitioning boundaries.

The image partitioning process and the image reconstruction processaccording to the present invention may be performed on a projectedimage. In this case, the reconstruction process may be used to rearrangefaces in the image as well as pixels in the image. This may be apossible example when the image is partitioned into or constructed witha plurality of faces. The following example will be described, focusingon the case in which the image is partitioned into tiles on the basis ofa face unit.

SX,Y (S0,0 to S3,2) in Section 21A may correspond to S′U,V (S′0,0 toS′2,1) in Section 21 B (here, X and Y may be the same as or differentfrom U and V), and the reconstruction process may be performed in faceunits. For example, S2,1, S3,1, S0,1, S1,2, S1,1, and S1,0 may beassigned to S′0,0, S′1,0, S′2,0, S′0,1, S′1,1, and S′2,1 (facerearrangement). Also, S2,1, S3,1, and S0,1 may not be reconstructed(pixel rearrangement), and S1,2, S1,1, and S1,0 may be rotated by 90degrees and then reconstructed. This may be represented as shown inSection 21C. In Section 21C, horizontally laid symbols S1,0, S1,1, andS1,2 may be images that are horizontally laid in order to maintaincontinuity of an image.

The reconstruction of the faces may be implicitly or explicitlyprocessed depending on encoding/decoding settings. The implicitprocessing may be performed according to a predetermined rule inconsideration of the type (here, a 360-degree image) and characteristics(here, a projection format, etc.) of the image.

For example, for S′0,0 and S′1,0; S′1,0 and S′2,0; S′0,1 and S′1,1; andS′1,1 and S′2,1 in Section 21C, there is image continuity (orcorrelation) between both faces with respect to the face boundary, andSection 21C may be an example in which there is continuity between threeupper faces and three lower faces. While the image is divided into aplurality of faces through a projection process from the 3D space to the2D space and then packed for each region, the reconstruction may beperformed in order to increase image continuity between faces toefficiently reconstruct the faces. Such reconstruction of the faces maybe predetermined and processed.

Alternatively, the reconstruction process may be performed throughexplicit processing, and reconstruction information may be generated.

For example, when information (e.g., one of implicitly acquiredinformation and explicitly generated information) regarding a M×Nconstruction (e.g., 6×1, 3×2, 2×3, 1×6, and the like for CMP compact; inthis example, a 3×2 configuration is assumed) is checked through theregion-wise packing process, face reconstruction may be performedaccording to the M×N construction, and then information regarding theface reconstruction may be generated. For example, when faces arerearranged in an image, index information (or information regardinglocations in the image) may be assigned to each face. When pixels arerearranged in a face, mode information for reconstruction may beassigned.

The index information may be pre-defined as shown in Sections 18A to 18Cof FIG. 18 . In Sections 21A to 21C, SX,Y or S′U,V represents each faceusing location information (e.g., S[i][j]) indicating a width and aheight or using one piece of location information (e.g., S[i]; it isassumed that the location information is assigned in a raster scanorder, starting from an upper left face of the image), and an index ofeach face may be assigned thereto.

For example, when an index is assigned using the location informationindicating the width and the height, face index #2 may be assigned toS′0,0, face index #3 may be assigned to S′1,0, face index #1 may beassigned to S′2,0, face index #5 may be assigned to S′0,1, face index #0may be assigned to S′1,1, and face index #4 may be assigned to S′2,1, asshown in Section 21C. Alternatively, when an index is assigned using onepiece of location information, face index #2 may be assigned to S[0],face index #3 may be assigned to S[1], face index #1 may be assigned toS[2], face index #5 may be assigned to S[3], face index #0 may beassigned to S[4], and face index #4 may be assigned to S[5]. Forconvenience of description, in the following example, S′0,0 to S′2,1 maybe referred to as a to f. Alternatively, each face may be representedusing location information indicating the width and height of a pixel orblock unit on the basis of an upper left corner of the image.

For the packed image acquired through the image reconstruction process(or the region-wise packing process), the face scan order is the same asor different from the image scan order depending on reconstructionsettings. For example, when one scan order (e.g., raster scan) isapplied to an image shown in Section 21A, a, b, and c may have the samescan order, and d, e, and f may have different scan orders. For example,when the scan order for Section 21A or the scan order for a, b, and cfollows an order of (0,0) □ (1,0) □ (0,1) □ (1,1), the scan order ford,e, and f may follow an order of (1,0) □ (1,1) □ (0,0) □ (0,1). This maybe determined according to image reconstruction settings, and suchsetting may be applied even to other projection formats.

In the image partitioning process shown in Section 21 B, a tile may beindividually set as a face unit. For example, each of the faces a to fmay be set as a tile unit. Alternatively, a plurality of face units maybe set as a tile. For example, the faces a to c may be set as one tile,and the faces d to f may be set as one tile. The construction may bedetermined on the basis of face characteristics (e.g., continuitybetween faces, etc.), and unlike the above example, different tilesettings for faces may be possible.

The following is an example of partitioning information according to aplurality of image partitioning processes. In this example, it isassumed that partitioning information for a face is omitted, a unitother than a face is a tile, and the partitioning information isvariously processed.

As a first example, the image partitioning information may be acquiredon the basis of face information and may be implicitly omitted. Forexample, a face may be individually set as a tile, or a plurality offaces may be set as a tile. In this case, when at least one face is setas a tile, this may be determined according to a predetermined rule onthe basis of face information (e.g., continuity or correlation).

As a second example, the image partitioning information may beexplicitly generated irrespective of the face information. For example,when the partitioning information is generated using the number ofcolumns (here, num_tile_columns) and the number of rows (here,num_tile_rows) of the tile, the partitioning information may begenerated in a method of the above-described image partitioning process.For example, the number of columns of the tile may range from 0 to thewidth of the image or the width of the block (here, a unit acquired fromthe picture partitioning part), and the number of rows of the tile mayrange from 0 to the height of the image or the height of the block.Also, additional partitioning information (e.g., uniform_spacing_flag)may be generated. In this case, the boundary of the face and theboundary of the partitioning unit may or may not match each otherdepending on the partitioning settings.

As a third example, the image partitioning information may be explicitlygenerated on the basis of the face information. For example, when thepartitioning information is generated using the numbers of columns androws of the tile, the partitioning information may be generated on thebasis of the face information (here, the number of columns ranges from 0to 2, and the number of rows ranges from 0 to 1; since the configurationof the faces in the image is 3×2). For example, the number of columns ofthe tile may range from 0 to 2, and the number of rows of the tile mayrange from 0 to 1. Also, additional partitioning information (e.g.,uniform_spacing_flag) may not be generated. In this case, the boundaryof the face and the boundary of the partitioning unit may match eachother.

In some cases (the second example and the third example are assumed), ansyntax element of the partitioning information may be differentlydefined, or syntax element settings (e.g., binarization settings; whenthe range of a candidate group of a syntax element is limited and small,other binarization may be used) may be differently applied even thoughthe same syntax element is used. The above example has been describedfor some of various elements of the partitioning information. However,the present invention is not limited thereto, and it can be understoodthat other settings are possible according to whether the partitioninginformation is generated on the basis of the face information.

FIG. 22 is an example diagram in which an image packed or projected byCMP is partitioned into tiles.

In this case, it is assumed to have the same tile partitioningboundaries (W0 to W2, H0, and H1 are all activated) as those shown inSection 21A of FIG. 21 and have the same tile partitioning boundaries(W0, W1, and H0 are all activated) as those shown in Section 21 B ofFIG. 21 . When it is assumed that a region P indicates the entire imageand a region V indicates a viewport, full decoding or partial decodingmay be performed. This example will be described, focusing on partialdecoding. In Section 22A, tiles e, f, and g may be decoded for CMP (aleft side) and tiles of a, c, and e may be decoded for CMP compact (aright side) to acquire a region corresponding to the viewport. InSection 22B, tiles b, f, and i may be decoded for CMP and tiles of d, e,and f may be decoded for CMP compact to acquire a region correspondingto the viewport.

The above example has been described for a case in which thepartitioning of a slice, a tile, or the like is performed on the basisof a face unit (or a face boundary). However, as shown in Section 20A ofFIG. 20 , the partitioning may be performed on the inside of a face(e.g., an image is composed of one face in ERP and composed of aplurality of faces in other projection format), or the partitioning maybe performed on the boundary of the face as well as the inside.

FIG. 23 is a conceptual diagram illustrating an example of resizing a360-degree image according to an embodiment of the present invention. Inthis case, it is assumed that an image is projected by ERP. Also, thefollowing example will be described, focusing on the case of expansion.

The projected image may be resized through a scale factor or through anoffset factor depending on an image resizing type. Here, an image beforeresizing may be P_Width×P_Height, and an image after resizing may beP′_Width×P′_Height.

For the scale factor, after the width and height of the image areresized through scale factors (here, a in width and b in height), thewidth (P_Width×a) and the height (P_Height×b) of the image may beacquired. For the offset factor, after the width and height of the imageare resized through offset factors (here, L and R in width and T and Bin height), the width (P_Width+L+R) and the height (P_Height+T+B) of theimage may be acquired. The resizing may be performed using apredetermined method, or the resizing may be performed using one methodselected from among a plurality of methods.

The data processing method in the following example will be described,focusing on the case of an offset factor. For the offset factor, as thedata processing method, there may be a filling method by using apredetermined pixel value, a filling method by copying outer pixels, afilling method by copying a specific region of an image, a fillingmethod by transforming a specific region of an image, and the like.

A 360-degree image may be resized in consideration of characteristics inwhich continuity is present at a boundary of the image. For ERP, anouter boundary is not present in the 3D space, but may be present whenthe 3D space is transformed into the 2D space through the projectionprocess. Data in a boundary region includes data with outwardcontinuity, but may have a boundary in terms of spatial characteristics.The resizing may be performed in consideration of such characteristics.In this case, the continuity may be checked according to the projectionformat or the like. For example, an ERP image may be an image havingcharacteristics in which both end boundaries are continuous. Thisexample will be described, assuming that left and right boundaries ofthe image are continuous with each other and upper and lower boundariesof the image are continuous with each other. The data processing methodwill be described, focusing on a filling method by copying a specificregion of the image and a filling method by transforming a specificregion of the image.

When the image is resized to the left, a resized region (here, LC orTL+LC+BL) may be filled with data of a right region (here, tr+rc+br) ofthe image having continuity with the left of the image. When the imageis resized to the right, a resized region (here, RC or TR+RC+BR) may befilled with data of a left region (here, tl+lc+bl) of the image havingcontinuity with the right of the image. When the image is resizedupward, a resized region (here, TC or TL+TC+TR) may be filled with dataof a lower region (here, bl+bc+br) of the image having continuity withthe upper side. When the image is resized downward, a resized region(here, BC or BL+BC+BR) may be filed with data.

When the size or length of the resized region is m, the resized regionmay have a range from (−m,y) to (−1,y) (resizing to the left) or a rangefrom (P_Width, y) to (P_Width+m−1,y) (resizing to the right) withrespect to coordinates of the image before resizing (here, x ranges from0 to P_Width−1). The location x′ of the region for acquiring the data ofthe resized region may be derived from a formula x′=(x+P_Width) %P_Width. In this case, x denotes a coordinate of a resized region withrespect to coordinates of an image before resizing, and x′ denotes acoordinate of a region referenced to a resized region with respect tocoordinates of an image before resizing. For example, when the image isresized to the left, m is 4, and the width of the image is 16,corresponding data of (−4,y) may be acquired from (12,y), correspondingdata of (−3,y) may be acquired from (13,y), corresponding data of (−2,y)may be acquired from (14,y), and corresponding data of (−1,y) may beacquired from (15,y). Alternatively, when the image is resized to theright, m is 4, and the width of the image is 16, corresponding data of(16,y) may be acquired from (0,y), corresponding data of (17,y) may beacquired from (1,y), corresponding data of (18,y) may be acquired from(2,y), and corresponding data of (19,y) may be acquired from (3,y).

When the size or length of the resized region is n, the resized regionmay have a range from (x,−n) to (x,−1) (resizing upward) or a range from(x,P_Height) to (x,P_Height+n−1) (resizing downward) with respect tocoordinates of the image before resizing (here, y ranges from 0 toP_Height−1). The location (y′) of the region for acquiring data of theresized region may be derived from a formula y′=(y+P_Height) % P_Height.In this case, y denotes a coordinate of a resized region with respect tocoordinates of an image before resizing, and y′ denotes a coordinate ofa region referenced to a resized region with respect to coordinates ofan image before resizing. For example, when the image is resized upward,n is 4, and the height of the image is 16, corresponding data of (x,−4)may be acquired from (x,12), corresponding data of (x,−3) may beacquired from (x,13), corresponding data of (x,−2) may be acquired from(x,14), and corresponding data of (x,−1) may be acquired from (x,15).Alternatively, when the image is resized downward, n is 4, and theheight of the image is 16, corresponding data of (x,16) may be acquiredfrom (x,0), corresponding data of (x,17) may be acquired from (x,1),corresponding data of (x,18) may be acquired from (x,2), andcorresponding data of (x,19) may be acquired from (x,3).

After the resized region is filled with data, the resizing may beperformed with respect to the coordinates of the image after resizing(here, x ranges from 0 to P′_Width−1, and y ranges from 0 toP′_Height−1). The example may be applied to a coordinate system oflatitude and longitude.

Various resizing combinations may be provided as follows.

As an example, the image may be resized to the left by m. Alternatively,the image may be resized to the right by n. Alternatively, the image maybe resized upward by o. Alternatively, the image may be resized downwardby p.

As an example, the image may be resized to the left by m and to theright by n. Alternatively, the image may be resized upward by o anddownward by p.

As an example, the image may be resized to the left by m, to the rightby n, and upward by o. Alternatively, the image may be resized to theleft by m, to the right by n, and downward by p. Alternatively, theimage may be resized to the left by m, upward by o, and downward by p.Alternatively, the image may be resized to the right by n, upward by o,and downward by p.

As an example, the image may be resized to the left by m, to the rightby n, upward by o, and downward by p.

Like the above example, at least one resizing operation may beperformed. Image resizing may be implicitly performed according toencoding/decoding settings, or resizing information may be implicitlygenerated and then image resizing may be performed on the basis of thegenerated resizing information. That is, m, n, o, and p of the aboveexample may be determined as predetermined values or may be explicitlygenerated using the resizing information. Alternatively, some may bedetermined as predetermined values, and the others may be explicitlygenerated.

The above example has been described, focusing on the case of data beingacquired from specific regions of the image, but other methods may alsobe applied. The data may be a pixel before encoding or a pixel afterencoding and may be determined according to characteristics of aresizing step or an image to be resized. For example, the data may referto an input pixel of a projected image, a packed image, or the like whenthe resizing is performed in a pre-processing process and a pre-encodingstep, and the data may refer to a restored pixel when the resizing isperformed in a post-processing process, an intra-prediction referencepixel generation step, a reference picture generation step, a filteringstep, and the like. Also, the resizing may be performed by individuallyusing a data processing method in each resized region.

FIG. 24 is a conceptual diagram illustrating continuity between faces ina projection format (e.g., CHP, OHP, or ISP) according to an embodimentof the present invention.

In detail, FIG. 24 may show an example of an image composed of aplurality of faces. The continuity may be a characteristic generated inadjacent regions in a 3D space. Sections 24A to 24C distinctly show acase A of having both of spatial adjacency and continuity whentransformation is made to a 2D space through a projection process, acase B of having spatial adjacency but no continuity, a case C of havingno spatial adjacency but continuity, and a case D of having neither ofspatial adjacency and continuity. Unlike this, general images areclassified into a case A of having both of spatial adjacency andcontinuity and a case D of having neither of spatial adjacency andcontinuity. In this case, the case of having continuity corresponds tosome of the examples (A or C).

That is, referring to Sections 24A to 24C, the case of having both ofspatial adjacency and continuity (here, which is described withreference to Section 24A) may be shown as b0 to b4, and the case havingno spatial adjacency but continuity may be shown as B0 to B6. That is,the cases indicate regions being adjacent in the 3D space, and it ispossible to enhance encoding performance by using characteristics inwhich b0 to b4 and B0 to B6 have continuity in an encoding process.

FIG. 25 is a conceptual diagram illustrating face continuity in Section21C which is an image acquired through an image reconstruction processor a region-wise packing process in the CMP projection format.

Here, Section 21C of FIG. 21 shows a rearrangement of a 360-degree imagespread in the shape of a cube in Section 21A, and thus face continuityapplied to Section 21A of FIG. 21 is maintained. That is, as shown inSection 25A, a face S2,1 may be horizontally continuous with faces S1,1and S3,1 and may be vertically continuous with a face S1,P rotated by 90degrees and a face S1, 2 rotated by −90 degrees.

In the same manner, the continuity of faces S3,1, S0,1, S1,2, S1,1, andS1,0 may be checked in Sections 25B to 25F.

Continuity between faces may be defined according to projection formatsettings or the like. However, the present invention is not limitedthereto, and modifications may be made thereto. The following examplewill be described on the assumption that continuity is present as shownin FIGS. 24 and 25 .

FIG. 26 is an example diagram illustrating image resizing in the CMPprojection format according to an embodiment of the present invention.

Section 26A shows an example of resizing an image, Section 26B shows anexample of resizing a face unit (or a partitioning unit), and Section26C shows an example of resizing an image and a face unit (or an exampleof performing multiple resizing).

The projected image may be resized through a scale factor or through anoffset factor depending on an image resizing type. Here, an image beforeresizing may be P_Width×P_Height, an image after resizing may beP′_Width×P′_Height, and the size of a face may be F_Width×F_Height. Thesize may be the same or different depending on the face, and the widthand height may be the same or different depending on the face. However,for convenience of description, this example will be described on theassumption that all faces in the image have the same size and the shapeof a square. Also, the description assumes that resizing values (here,WX and HY) are the same. In the following example, a data processingmethod will be described, focusing on the case of an offset factor andalso focusing on a filling method by copying a specific region of theimage and a filling method by transforming a specific region of theimage. The above settings may be applied even to the case shown in FIG.27 .

For Sections 26A to 26C, a boundary of a face may have continuity with aboundary of another face (here, it is assumed to have continuitycorresponding to Section 24A of FIG. 24 ). Here, the continuity may beclassified into a case of having spatial adjacency and image continuityin the 2D plane (a first example) and a case of having no spatialadjacency but image continuity in the 2D plane (a second example).

For example, when the continuity in Section 24A of FIG. 24 is assumed,upper, left, right, and lower regions of S1,1 may be spatially adjacentto, and have image continuity with, lower, right, left, and upperregions of S1,0, S0,1, S2,1, and S1,2 (the first example).

Alternatively, the left and right regions of S1,0 are not spatiallyadjacent to, but may have image continuity with, the upper regions ofS0,1 and S2,1 (the second example). Also, the left region of S0,1 maynot be spatially adjacent to, but have image continuity with, each other(the second example). Also, the left region and the right region of S1,2may be continuous with the lower regions of S0,1 and S2,1 (the secondexample). This may be merely a limited example, and other configurationsmay be applied depending on the definition and settings of theprojection format. For convenience of description, S0,0 to S3,2 inSection 26A are referred to as a to l.

Section 26A may be an example of a filling method using data of a regionhaving continuity toward an outer boundary of an image. A range from aregion A, which includes no data, to a resized region (here, a0 to a2,c0, d0 to d2, i0 to i2, k0, and l0 to l2) may be filled with anypredetermined value or through outer pixel padding, and a range from aregion B, which includes actual data, to a resized region (here, b0, e0,h0, and j0) may be filled with data of a region (or a face) having imagecontinuity. For example, b0 may be filled with data of an upper side ofthe face h, e0 may be filled with data of a right side of the face h, h0may be filled with data of a left side of the face e, and j0 may befilled with data of a lower side of the face h.

In detail, as an example, b0 may be filled with data of a lower side ofa face acquired by rotating the face h by 180 degrees, and j0 may befilled with data of an upper side of a face acquired by rotating theface h by 180 degrees. However, this example (including the followingexample) may represent only the location of a reference face, and dataacquired from the resized region may be acquired after a resizingprocess (e.g., rotation, etc.) that considers continuity between facesas shown in FIGS. 24 and 25 .

Section 26B may be an example of a filling method using data of a regionhaving continuity toward an inner boundary of an image. In this example,a different resizing operation may be performed for each face. Areduction process may be performed in the region A, and an expansionprocess may be performed in the region B. For example, the face a may beresized (here, reduced) to the right by w0, and the face b may beresized (here, expanded) to the left by w0. Alternatively, the face amay be resized (here, reduced) downward by h0, and the face e may beresized (here, expanded) upward by h0. In this example, when a change inwidth of the image is viewed through the faces a, b, c, and d, the facea is reduced by w0, the face b is expanded by w0 and w1, and the face cmay be reduced by w1. Thus, the width of the image before resizing isthe same as the width of the image after resizing. When a change inheight of the image is viewed through the faces a, e, and i, the face ais reduced by h0, the face e is expanded by h0 and h1, and the face imay be reduced by h1. Thus, the height of the image before resizing isthe same as the height of the image after resizing.

The resized regions (here, b0, e0, be, b1, bg, g0, h0, e1, ej, j0, gi,g1, j1, and h1) may be simply removed in consideration that the regionsare reduced from the region A which does not include data, and may befilled with data of a region having continuity in consideration that theregions are expanded from the region B which includes actual data.

For example, b0 may be filled with data of an upper side of the face e;e0 may be filled with data of a left side of the face b; be may befilled with data of a left side of the face b, an upper side of the facee, or a weighted sum of a left side of the face b and an upper side ofthe face e; b1 may be filled with data of an upper side of the face g;bg may be filled with data of a left side of the face b, an upper sideof the face g, or a weighted sum of a right side of the face b and anupper side of the face g; g0 may be filled with data of a right side ofthe face b; h0 may be filled with data of an upper side of the face b;e1 may be filled with data of a left side of the face j; ej may befilled with data of a lower side of the face e, a left side of the facej, or a weighed sum of a lower side of the face e and a left side of theface j; j0 may be filled with data of a lower side of the face e; gj maybe filled with data of a lower side of the face g, a left side of theface j, or a weighted sum of a lower side of the face g and a right sideof the face j; g1 may be filled with data of a right side of the face j;j1 may be filled with data of a lower side of the face g; and h1 may befilled with data of a lower side of the face j.

In the above example, when the resized region is filled with data ofspecific regions of the image, data of a corresponding region may becopied and then used to fill the resized region or may be transformed onthe basis of the characteristics, type, and the like of the image andthen used to fill the resized region. For example, when a 360-degreeimage may be transformed into the 2D space according to a projectionformat, a coordinate system (e.g., a 2D planar coordinate system) may bedefined for each face. For convenience of description, it is assumedthat (x, y, z) in the 3D space is transformed into (x,y,C), (x,C,z), or(C,y,z) for each face. The above example indicates a case in which, froma resized region of a face, data of a face other than the correspondingface is acquired. That is, when the resizing is performed on the currentface, data of another face with different coordinate systemcharacteristics may be copied as it is and then used. In this case,there is a possibility that the continuity is distorted based on theresizing boundary. To this end, data of another face acquired accordingto coordinate system characteristics of the current face may betransformed and used to fill a resized region. The transformation isalso merely an example of the data processing method, and the presentinvention is not limited thereto.

When data of specific regions of the image is copied and used to fill aresized region, distorted continuity (or radically changing continuity)may be included in a boundary region between a resized region e and aresized region e0. For example, the continuity may change with respectto a boundary, and a straight line edge may be curved with respect tothe boundary.

When data of specific regions of the image is transformed and used tofill a resized region, gradually changing continuity may be included ina boundary region between resized regions.

The above example may be an example of the data processing method of thepresent invention to transform data of specific regions of the image onthe basis of the characteristics, type, and the like of the image andfill a resized region with the transformed data.

Section 26C may be an example of filling a resized region with data of aregion having continuity toward boundaries (an inner boundary and anouter boundary) of the image in combination of the image resizingprocesses corresponding to Sections 26A and 26B. The resizing process ofthis example may be derived from those of Sections 26A and 26B, and adetailed description thereof will be omitted.

Section 26A may be an example of the process of resizing an image, andSection 26B may be an example of resizing a partitioning unit in animage. Section 26C may be an example of a plurality of resizingprocesses including the process of resizing an image and the process ofresizing a partitioning unit in an image.

For example, an image (here, a first format) acquired through aprojection process may be resized (here, a region C), and an image(here, a second format) acquired through a format transformation processmay be resized (here, a region D). In this example, an image projectedby ERP may be resized (here, a full image) and transformed into an imageprojected by CMP through a format transformation part, and the imageprojected by CMP may be resized (here, a face unit). The above exampleis an example in which a plurality of resizing operations are performed.However, the present invention is not limited thereto, and modificationsmay be made thereto.

FIG. 27 is an example diagram illustrating resizing of an imagetransformed and packed in the CMP projection format according to anembodiment of the present invention. FIG. 27 also assumes continuitybetween faces as shown in FIG. 25 , and thus the boundary of a face mayhave continuity with the boundary of another face.

In this example, offset factors of W0 to W5 and H0 to H3 may havevarious values (here, it is assumed that the offset factors are used asresizing values). For example, the offset factors may be derived from apredetermined value, a motion search range of inter-prediction, a unitacquired from a picture partitioning part, and the like, and other casesare also possible. In this case, the unit acquired from the pixelpartitioning unit may include a face. That is, the resizing values maybe determined on the basis of F_Width and F_Height.

Section 27A is an example of individually resizing a single face (here,upward, downward, to the left, and to the right with respect to theface) and filling expanded regions with data of a region havingcontinuity. For example, outer regions a0 to a6 of the face a may befilled with continuous data, and outer regions b0 to b6 of the face bmay be filled with continuous data.

Section 27B is an example of resizing a plurality of faces (here,upward, downward, to the left, and to the right with respect to theplurality of faces) and filling expanded regions with data of a regionhaving continuity. For example, the faces a, b, and c may be expanded tothe outer regions a0 to a4, b0 and b1, and c0 to c4.

Section 27C may be an example of resizing a full image (here, upward,downward, to the left, and to the right with respect to the full image)and filling expanded regions with data of a region having continuity.For example, a full image composed of the faces a to f may be expandedto the outer regions a0 to a2, b0, c0 to c2, d0 to d2, and f0 to f2.

That is, the resizing may be performed in a single face unit, in aplurality of face units having continuity with one another, and in afull image unit.

In the above example, the resized regions (here, a0 to f7) may be filledwith data of a region (or a face) having continuity, as shown in Section24A. That is, the resized regions may be filled with data of uppersides, lower sides, left sides, and right sides of the faces a to f.

FIG. 28 is an example diagram illustrating a data processing method forresizing a 360-degree image according to an embodiment of the presentinvention.

Referring to FIG. 28 , a region B (a0 to a2, ad0, b0, c0 to c2, cf1, d0to d2, e0, f0 to f2), which is a resized region, may be filled with dataof a region having continuity among pixel data belonging to a to f.Also, a region C (ad1, be, cf0), which is another resized region), maybe filled with data of a region to be resized and data of a regionhaving spatial adjacency but no continuity in combination.Alternatively, since the resizing is performed between two regions(e.g., a and d, b and e, and c and f) selected from among a to f, theregion C may be filled with pieces of data of the two regions incombination. For example, the face b and face e may be spatiallyadjacent to each other, but have no continuity with each other. Aresized region be located between the face b and the face e may beresized using data of the face b and data of the face e. For example,the region be may be filled with a value acquired by averaging the dataof the face b and the data of the face e or with a value acquiredthrough a distance-based weighted sum. In this case, a pixel that isused for data to be used to fill a resized region in the face b and theface e may be a boundary pixel for each face or an internal pixel ofeach face.

In summary, a resized region between partitioning units of the image maybe filled with data generated by using pieces of data of the two unitsin combination.

The data processing method may be supported in some conditions (here,when a plurality of regions are resized.

In Sections 27A and 27B, a region to be resized between partitioningunits is constructed individually for each partitioning unit (in Section27A, a6 and d1 are constructed for a and d, respectively). In FIG. 28 ,a single region to be resized between partitioning units may beconstructed for adjacent partitioning units (ad1 is constructed for aand d). It will be appreciated that the method may be included in thecandidate group for the data processing method in Sections 27A and 27Band the resizing may be performed using a data processing methoddifferent from the above example even in FIG. 28 .

In the process of resizing an image according to the present invention,a predetermined data processing method may be implicitly used in aresized region, or one of a plurality of data processing methods may beused to explicitly related information. The predetermined dataprocessing method may be one of a filling method by using any pixelvalue, a filling method by copying outer pixels, a filling method bycopying a specific region of an image, a filling method by transforminga specific region of an image, a filling method using data derived froma plurality of regions of an image, etc. For example, when a resizedregion is located inside an image (e.g., a packed image) and regions atboth sides (e.g., a face) have spatial adjacency but no continuity, adata processing method may be applied to fill the resized region withdata derived from a plurality of regions. Also, the resizing may beperformed by one data processing method selected from among theplurality of data processing method, and related selection informationmay be explicitly generated. This may be an example applicable to ageneral image as well as a 360-degree image.

The encoder may add the information generated during the above processto a bitstream in units of at least one of sequences, pictures, slices,tiles, and the like, and the decoder may parse related information fromthe bitstream. Also, the information may be included in the bitstream inthe form of SEI or metadata. The partitioning process, thereconstruction process, and the resizing process for a 360-degree imagehave been described, focusing on some projection formats such as ERP andCMP. However, the present invention is limited thereto, and theabove-description may be applied even to other projection formats as itis or after modified.

It has been described that the image setting process for the above360-degree image encoding/decoding apparatus may be applied to apre-processing process, a post-processing process, a formattransformation process, an inverse format transformation process, andthe like as well as encoding/decoding processes.

In summary, the projection process may be constructed to include imagesetting processes. In detail, the projection process may be performed inaddition to at least one of the image setting processes. Thepartitioning may be performed in units of regions (or faces) on thebasis of the projected image. Depending on the projection format, thepartitioning may be performed on a single region or a plurality ofregions. For the partitioning, partitioning information may begenerated. Also, the projected image may be resized, or a projectedregion may be resized. In this case, the resizing may be performed on atleast one region. For the resizing, resizing information may begenerated. Also, the projected image may be reconstructed (orface-arranged), or a projected region may be reconstructed. In thiscase, the reconstruction may be performed on at least one region. Forthe reconstruction, reconstruction information may be generated.

In summary, a region-wise packing process may be constructed to includeimage setting processes. In detail, the region-wise packing projectionprocess may be performed in addition to at least one of the imagesetting processes. The partitioning process may be performed in units ofregions (or faces) on the basis of the packed image. Depending on theregion-wise packing settings, the partitioning may be performed on asingle region or a plurality of regions. For the partitioning,partitioning information may be generated. Also, the packed image may beresized, or a packed region may be resized. In this case, the resizingmay be performed on at least one region. For the resizing, resizinginformation may be generated. Also, the packed image may bereconstructed, or a packed region may be reconstructed. In this case,the reconstruction may be performed on at least one region. For thereconstruction, reconstruction information may be generated.

During the projection process, all or some of the image settingprocesses may be performed, and image setting information may beincluded. This information may be setting information for the projectedimage. In detail, this information may be setting information forregions in the projected image.

During the region-wise packing process, all or some of the image settingprocesses may be performed, and image setting information may beincluded. This information may be setting information for the packedimage. In detail, this information may be setting information forregions in the packed image. Alternatively, this information may bemapping information (e.g., see the description with reference to FIG. 11; this can be understood assuming that P0 and P1 indicate projectedimages and S0 to S5 indicate packed images) between the projected imageand the packed image. In detail, this information may be mappinginformation between a specific region in the projected image and aspecific region in the packed image. That is, this information may besetting information assigned from the specific region in the projectedimage to the specific region in the packed image.

The image information may be represented as information acquired throughthe above-described various embodiments during the image setting processof the present invention. For example, when related information isrepresented using at least one syntax element in Table 1 to Table 6, thesetting information for the projected image may includepic_width_in_samples, pic_height_in_samples, part_top[i], part_left[i],part_width[i], part_height[i], and the like, and the setting informationfor the packed image may include pic_width_in_samples,pic_height_in_samples, part_top[i], part_left[i], part_width[i],part_height[i], convert_type_flag[i], part_resizing_flag[i],top_height_offset[i], bottom_height_offset[i], left_width_offset[i],right_width_offset[i], resizing_type_flag[i], and the like. The aboveexample may be an example of explicitly generating information regardingfaces (e.g., part_top[i], part_left[i], part_width[i], andpart_height[i] among the setting information of the projected image).

Some of the image setting process may be included in a projectionprocess or a region-wise packing process corresponding to the projectionformat through a predetermined operation.

For example, ERP uses a method of filling regions expanded to the leftby m and to the right by n with data of regions in directions oppositeto the resizing directions for the image, and thus the resizing processmay be implicitly included. Alternatively, CMP uses a method of fillingregions expanded upward by m, downward by n, to the left by o, and tothe right by p with data of a region having continuity with a resizedregion, and thus the resizing process may be implicitly included.

In the above example, the projection format may be an example ofsubstitute formats capable of replacing the conventional projectionformats or an example of additional formats (e.g., ERP1 and CMP1) forthe conventional projection formats. However, the present invention isnot limited thereto, examples of various image setting processes of thepresent invention may be alternatively combined, and similarapplications may be possible for other formats.

Although not shown in the image encoding apparatus and the imagedecoding apparatus of FIGS. 1 and 2 , a block partitioning part may befurther included. Information regarding a default encoding part may beacquired from the picture partitioning part, and default encoding partmay refer to a default (or start) unit for prediction, transformation,quantization, etc. during the image encoding/decoding process. In thiscase, the encoding part may be composed of one luminance encoding blockand two chrominance encoding blocks according to a color format (here,YCbCr), and the size of the blocks may be determined according to thecolor format. The following example will be described with respect tothe blocks (here, a luminance component). In this case, it is assumedthat a block is a unit that may be acquired after each unit isdetermined, and it is also assumed that similar settings are applicableto other types of blocks.

The block partitioning part may be set in association with each elementof the image encoding apparatus and the image decoding apparatus.Through this process, the size and shape of the blocks may bedetermined. In this case, a different block may be defined for eachelement. The block may be a prediction block for the prediction part, atransformation block for the transformation part, a quantization blockfor the quantization part, or the like. However, the present inventionis not limited thereto, and an additional block unit may be defined foranother element. The size and shape of the block may be defined by thewidth and height of the block.

A block may be expressed as M×N by the block partitioning part and maybe acquired in the range from a minimum value to a maximum value. Forexample, when a block supports a square shape and has a maximum value of256×256 and a minimum value of 8×8, a block having a size of 2^(m)×2^(m)(here, m is an integer from 3 to 8; for example, 8×8, 16×16, 32×32,64×64, 128×128, and 256×256), a block having a size of 2m×2m (here, m isan integer from 4 to 128), or a block having a size of m×m (here, m isan integer from 8 to 128) may be acquired. Alternatively, when a blocksupports square and rectangle forms and has the same range as describedabove, a block having a size of 2^(m)×2^(n) (here, m and n are integersfrom 3 to 8; when it is assumed that the maximum aspect ratio is 2:1,8×8, 8×16, 16×8, 16×16, 16×32, 32×16, 32×32, 32×64, 64×32, 64×64,64×128, 128×64, 128×128, 128×256, 256×128, 256×256; there may be nolimitation on the aspect ratio, or the maximum aspect ratio may bepresent depending on encoding/decoding settings) may be acquired.Alternatively, a block having a size of 2m×2n (here, m and n areintegers from 4 to 128) may be acquired. Alternatively, a block having asize of m×n (here, m and n are integers from 8 to 256) may be acquired.

Acquirable blocks may be determined according to encoding/decodingsettings (e.g., a block type, a partitioning scheme, a partitioningsetting, etc.). For example, a block with a size of 2^(m)×2^(n) may beacquired as the encoding block, a block with a size of 2m×2n or m×n maybe acquired as the prediction block, and a block with a size of 2m×2^(n)may be acquired as the transformation block. Information regarding thesize and range of the blocks (e.g., information related to an exponentand a multiple) may be generated on the basis of the settings.

The range (here, which is determined using the maximum value and theminimum value) may be determined depending on the block type. Also, someblocks may have block range information explicitly generated, and otherblocks may have block range information implicitly determined. Forexample, the encoding block and the transformation block may haverelated information explicitly generated, and the prediction block mayhave related information implicitly processed.

In the explicit case, at least one piece of range information may begenerated. For example, the range information of the encoding block maybe generated as information regarding the maximum value and the minimumvalue. Alternatively, the range information may be generated on thebasis of a difference (e.g., which is generated based on the settings;index difference information between the minimum value and the maximumvalue, etc.) between the predetermined minimum value (e.g., eight) andthe maximum value. Also, a plurality of pieces of range information forthe width and height of a rectangular block may be generated.

In the implicit case, the range information may be acquired on the basisof encoding/decoding settings (e.g., a block type, a partitioningscheme, a partitioning setting, etc.). For example, for the predictionblock, the encoding block (here, having a maximum size of M×N and aminimum size of m×n), which is an upper unit, may acquire informationregarding the maximum value and the minimum value according to acandidate group (here, M×N and m/2×n/2) acquirable from partitioningsettings (here, quad-tree partitioning+partitioning depth of 0) of theprediction block.

The size and shape of an initial (or start) block of the blockpartitioning part may be determined from its upper unit. The initialblock of the encoding block may be the default encoding block acquiredfrom the picture partitioning part, the initial block of the predictionblock may be the encoding block, and the initial block of thetransformation block may be the encoding block or the prediction block,which may be determined according to encoding/decoding settings. Forexample, the prediction block is an upper unit of the transformationblock when the encoding mode is an intra mode, and the prediction blockis a unit that is independent of the transformation block when theencoding node is an inter mode. The initial block, which is a startblock for partitioning, may be partitioned into small blocks. When anoptimal size and shape corresponding to the block partitioning aredetermined, the block may be determined as an initial block of a lowerunit. For example, the former case may correspond to the encoding block,and the latter case (lower unit) may correspond to the prediction blockor the transformation block. As described above, when the initial blockof the lower unit is determined, a partitioning process for finding ablock of the optimal size and shape may be performed.

In summary, the block partitioning part may partition the defaultencoding unit (or the maximum encoding unit) into at least one encodingunit (or lower encoding unit). Also, the encoding unit may bepartitioned into at least one prediction unit and also may bepartitioned into at least one transformation unit. The encoding unit maybe partitioned into at least one encoding block, and the encoding blockmay be partitioned into at least one prediction block and also may bepartitioned into at least one transformation block. Also, the predictionunit may be partitioned into at least one prediction block, and thetransformation unit may be partitioned into at least one transformationblock.

When the block of the optimal size and form is found through the modedetermination process as described above, mode information related tothe block (e.g., partitioning information, etc.) may be generated. Themode information may be added to a bitstream in addition to informationgenerated in a construction unit to which the block belongs (e.g.,prediction-related information and transformation-related information)and then transmitted to a decoder. The mode information may be parsed bythe decoder at units of the same level and then used during an imagedecoding process.

The following example will describe a partitioning scheme and assumethat the initial block has the shape of a square. However, the same orsimilar applications may be possible for rectangular shapes.

The block partitioning part may support various partitioning schemes.For example, the block partitioning part may support tree-basedpartitioning or type-based partitioning, and other methods may beapplied thereto. The tree-based partitioning may generate partitioninginformation with partitioning flags, and the type-based partitioning maybe generate partitioning information with index information for blockforms included in a predetermined candidate group.

FIG. 29 is an example diagram showing a tree-based block form.

Section 29A shows a single 2N×2N block that is not partitioned, Section29B shows an example in which two 2N×N blocks are acquired through somepartitioning flags (here, binary tree-based horizontal partitioning),Section 29C shows an example in which two N×2N blocks are acquiredthrough some partitioning flags (here, binary tree-based verticalpartitioning), and Section 29D shows an example in which four N×N blocksare acquired through some partitioning flags (here, quad-treepartitioning or horizontal and vertical binary-tree partitioning). Theacquired block form may be determined according to the type of the treeused for the partitioning. For example, when the quad-tree partitioningis performed, acquirable candidate blocks may correspond to Sections 29Aand 29D. When the binary-tree partitioning is performed, acquirablecandidate blocks may correspond to Sections 29A, 29B, 29C, and 29D. Thequad tree supports a single partitioning flag. The flag being “0” mayacquire Section 29A, and the flag being “1” may acquire Section 29D. Thebinary tree supports a plurality of partitioning flags. Among thepartitioning flags, one partitioning flag may be a flag indicatingwhether partitioning is performed, another partitioning flag may be aflag indicating whether partitioning is horizontal or vertical, andstill another partitioning flag may be a flag indicating whether toallow overlapping of horizontal/vertical partitioning. When theoverlapping is allowed, acquirable candidate blocks may correspond toSections 29A, 29B, 29C, and 29D. When the overlapping is not allowed,acquirable candidate blocks may correspond to Sections 29A, 29B, and29C. The quad tree may be a default tree-based partitioning scheme, andan additional tree partitioning scheme (here, a binary tree) may beincluded in the tree-based partitioning scheme. When a flag allowingadditional tree partitioning is implicitly or explicitly activated, aplurality of tree partitioning operations may be performed. Thetree-based partitioning may allow recursive partition. That is, thepartitioned block may be set as an initial block again, and thetree-based partitioning may be performed, which may be determinedaccording to partitioning settings such as a partitioning range, apartitioning allowable depth, etc. This may be an example hierarchicalpartitioning scheme.

FIG. 30 is an example diagram showing a type-based block form.

Referring to FIG. 30 , a block after type-based partitioning may have a1-partitioned form (here, Section 30A), a 2-partitioned form (here,Sections 30B, 30C, 30D, 30E, 30F, and 30G), and a 4-partitioned form(here, Section 30H). Candidates may be constructed through variousconstructions. For example, the candidates may be constructed as a, b,c, and n; a, b to g, and n; or a, n, and q of FIG. 31 . However, thepresent invention is not limited thereto, and various modifications maybe possible, including the following example. Blocks supported when aflag allowing symmetric partition is activated may correspond toSections 30A, 30B, 30C, and 30H, and blocks supported when a flagallowing asymmetric partition is activated may corresponding to all ofSections 30A to 30H. For the former case, related information (here, theflag allowing the symmetric partition) may be implicitly activated. Forthe latter case, related information (here, the flag allowing theasymmetric partition) may be explicitly generated. The tree-basedpartitioning may support one-time partitioning. Compared to thetree-based partitioning, a block acquired through the type-basedpartitioning may no longer be further partitioned. This may be anexample in which the partitioning allowable depth is 0.

FIG. 31 is an example diagram showing various types of blocks that maybe acquired by a block partitioning part of the present invention.

Referring to FIG. 31 , blocks in Sections 31A to 31S may be acquiredaccording to partitioning settings and partitioning schemes, additionalblock forms that are not shown may also be possible.

As an example, asymmetric partition may be allowed for the tree-basedpartitioning. For example, the binary tree may allow blocks shown inSections 31B and 31C (here, partitioning into a plurality of blocks) ormay allow blocks shown in Sections 30B to 31G (here, partitioning into aplurality of blocks). When the flag allowing asymmetric partition isexplicitly or implicitly deactivated according to encoding/decodingsettings, acquirable candidate blocks may be blocks in Section 31B or31C (here, assuming that overlapping of horizontal partitioning andvertical partitioning is not allowed). When the flag allowing asymmetricpartition is activated, acquirable candidate blocks may be blocks inSections 31 B, 31D, and 31E (here, horizontal partitioning) or blocks inSections 31C, 31F, and 31G (here, vertical partitioning). This examplemay correspond to a case in which a partitioning direction is determinedby a horizontal or vertical partitioning flag and a block form isdetermined according to a flag allowing asymmetry. However, the presentinvention is not limited thereto, and modifications may be made thereto.

As an example, additional tree partitioning may be allowed for thetree-based partitioning. For example, triple tree-based partitioning,quad-tree partitioning, octa tree-based partitioning, and the like maybe allowed, and thus n partitioning blocks (here, 3, 4, and 8; n is aninteger) may be acquired. Blocks supported for the triple tree-basedpartitioning may be blocks in Sections 31 H to 31M, blocks supported forthe quad-tree partitioning may be blocks in Sections 31N to 31Q, andblocks supported for the octa tree-based partitioning may be blocks inSection 31Q. Whether to support tree-based partitioning may beimplicitly determined according to encoding/decoding settings, andrelated information may be explicitly generated. Also, the binary-treepartitioning and the quad-tree partitioning may be used solely or incombination depending on encoding/decoding settings. For example, blocksas shown in Sections 31 B and 31C may be possible for the binary tree,and blocks as shown in Sections 31B, 31C, 31I, and 31L may be possiblewhen the binary tree and the triple tree are used in combination. Whenother than the existing flags, a flag allowing additional partitioningis explicitly or implicitly deactivated according to encoding/decodingsettings, acquirable candidate blocks may be blocks in Section 31B or31C. When the flag for allowing additional partitioning is activated,acquirable candidate blocks may be blocks in Sections 31B and 31I or inSections 31B, 31H, 31I, and 31J (here, horizontal partitioning) orblocks in Section 31C or 31L or in the 31C, 31K, 31L, and 31M (here,vertical partitioning). This example may correspond to a case in which apartitioning direction is determined by the horizontal or verticalpartitioning flag and a block form is determined according to the flagallowing additional partitioning. However, the present invention is notlimited thereto, and modifications may be made thereto.

As an example, non-rectangular partition may be allowed for thetype-based block. For example, the partitioning as shown in Sections 31Rand 31S may be possible. When the block is combined with the type-basedblock candidates, the blocks of Sections 31A, 31B, 31C, 31H, 31R, and31S or Sections 31A to 31H, 31R, and 31S may be supported. Also, a blockthat supports n-partitioning (e.g., n is an integer; here, 3 other than1, 2, and 4) as shown in Sections 31 H to 31M may be included in thecandidate group.

The partitioning scheme may be determined according to encoding/decodingsettings.

As an example, the partitioning scheme may be determined according to ablock type. For example, an encoding block and a transformation blockmay use the tree-based partitioning, and a prediction block may use thetype-based partitioning. Alternatively, the partitioning scheme may beused in combination thereof. For example, the prediction block may use apartitioning scheme obtained by using the tree-based partitioning andthe type-based partitioning in combination, and a partitioning schemebeing applied may differ depending on at least one range of the block.

As an example, the partitioning scheme may be determined according to ablock size. For example, the tree-based partitioning may be applied to aspecific range (e.g., from a×b to c×d; when the latter is greater)between the minimum value and the maximum value of the block, and thetype-based partitioning may be applied to another specific range (e.g.,from e×f to g×h). In this case, range information according to thepartitioning scheme may be explicitly generated or implicitlydetermined.

As an example, the partitioning scheme may be determined according tothe shape of a block (or a block before partitioning). For example, whenthe block has a square shape, the tree-based partitioning and thetype-based partitioning may be possible. Alternatively, when the blockhas a rectangular shape, the tree-based partitioning may be possible.

The partitioning settings may be determined according toencoding/decoding settings.

As an example, the partitioning settings may be determined according toa block type. For example, for the tree-based partitioning, an encodingblock and a prediction block may use the quad-tree partitioning, andblock a transformation block may use the binary-tree partitioning.Alternatively, the partitioning allowable depth of the encoding blockmay be set to m, the partitioning allowable depth of the predictionblock may be set to n, and the partitioning allowable depth of thetransformation block may be set to o. Here, m, n, and o may or may notbe the same.

As an example, the partitioning settings may be determined according toa block size. For example, the quad-tree partitioning may be applied toa specific range of a block (e.g., from a×b to c×d), and the binary-treepartitioning may be applied to another specific range (e.g., from e×f tog×h; here, it is assumed that c×d is greater than g×h). In this case,the range may include all ranges between the maximum value and theminimum value of the block, and the ranges may be set to overlap oneanother or not to overlap one another. For example, the minimum value ofa specific range may be equal to the maximum value of another specificrange, or the minimum value of a specific range may be smaller than themaximum value of another specific range. When there is an overlappingrange, a partitioning scheme with a greater maximum vale may have ahigher priority. That is, whether to perform a partitioning scheme witha lower priority may be determined according to a result of partitioningin the partitioning scheme with the higher priority. In this case, rangeinformation according to the tree type may be explicitly generated orimplicitly determined.

As another example, type-based partitioning with a specific candidategroup may be applied to the specific range of the block (which is thesame as the above example), and type-based partitioning with a specificcandidate group (which has at least one different configuration from theformer candidate group) may be applied to the specific range (which isthe same as the above example). In this case, the range may include allranges between the maximum value and the minimum value of the block, andthe ranges may be set not to overlap one another.

As an example, the partitioning settings may be determined according toa block shape. For example, the block has a square shape, the quad-treepartitioning may be possible. Alternatively, when the block has arectangular shape, the binary-tree partitioning may be possible.

As an example, the partitioning settings may be determined according toencoding/decoding information (e.g., a slice type, a color component, anencoding mode, etc.). For example, the quad-tree partitioning (or thebinary-tree partitioning) may be possible for a specific range (e.g.,from a×b to c×d) when the slice type is “I,” may be possible for aspecific range (e.g., from e×f to g×h) when the slice type is “P,” andmay be possible for a specific range (e.g., from i×j to k×l) when theslice type is “B.” Also, the partitioning allowable depth of thequad-tree (or the binary-tree) partitioning may be set to m when theslice type is “I,” may be set to n when the slice type is “P,” and maybe set to o when the slice type is “B.” Here, m, n, and o may or may notbe the same as one another. Some slice types may have the same settingsas the other slices (e.g., slice “P” and slice “B”).

As another example, the partitioning allowable depth of the quad tree(or the binary tree) may be set to m when the color component is aluminance component and may be set to n when the color component is achrominance component. Here, m and n may or may not be the same. Also,the range (e.g., from a×b to c×d) of the quad-tree (or binary-tree)partitioning when the color component is a luminance component and therange (e.g., from e×f to g×h) of the quad-tree (or binary-tree)partitioning when the color component is a chrominance component may ormay not be the same.

As another example, the partitioning allowable depth of the quad tree(or binary tree) may be m when the encoding mode is an intra mode, andmay be n when the encoding mode is an inter mode (here, it is assumedthat n is greater than m). Here, m and n may or may not be the same.Also, the range of the quad-tree (or binary-tree) partitioning when theencoding mode is the intra mode and the range of the quad-tree (orbinary-tree) partitioning when the encoding mode is the inter mode mayor may not be the same.

For the above example, information regarding whether to adaptivelysupport partitioning candidate group elements may be explicitlygenerated or implicitly determined according to encoding/decodinginformation.

A case in which the partitioning scheme and the partitioning settingsare determined according to encoding/decoding settings has beendescribed through the above example. The above example may show somecases for each element, and modifications may be made to other cases.Also, the partitioning scheme and the partitioning settings may bedetermined according to a combination of a plurality of elements. Forexample, the partitioning scheme and the partitioning settings may bedetermined by a block type, a block size, a block form,encoding/decoding information, etc.

Also, in the above example, elements involved in the partitioningscheme, settings, etc. may be implicitly determined and information maybe explicitly generated to determine whether to allow an adaptive casesuch as the above example.

Among the partitioning settings, a partitioning depth refers to thenumber of times an initial block is spatially partitioned (here, thepartitioning depth of the initial block is 0). As the partitioning depthincreases, the size of blocks into which the initial block ispartitioned may decrease. Thus, depth-related settings may differdepending on the partitioning scheme. For example, one common depth maybe used for the partitioning depth of the quad tree and the partitioningdepth of the binary tree among the tree-based partitioning schemes, adepth may be used individually depending on a tree type.

When in the above example, a partitioning depth is used individuallyaccording to a tree type, the partitioning depth at a partitioning startposition of the tree (here, a block before the partitioning) may be setto 0. The partitioning depth may be calculated not on the basis of thepartitioning range (here, the maximum value) of each range but focusingon the partitioning start position.

FIG. 32 is an example diagram illustrating tree-based partitioningaccording to an embodiment of the present invention.

Section 32A shows examples of quad-tree partitioning and binary-treepartitioning. In detail, in Section 32A, an upper left block shows anexample of quad-tree partitioning, upper right and lower left blocksshow example of quad-tree partitioning and binary-tree partitioning, anda lower right block shows an example of binary-tree partitioning. In thedrawings, a solid line (here, Quad1) represents a boundary line forquad-tree partitioning, a dotted line (here, Binary1) represents aboundary line for binary-tree partitioning, and a thick solid linerepresents a boundary line for binary-tree partitioning. A differencebetween the dotted line and the solid line may indicate a partitioningscheme difference.

As an example, (assuming the following conditions: the upper left blockhas a quad-tree partitioning allowable depth of 3; when a current blockis N×N, partitioning is performed until one of the width and the heightreaches N>>3, partitioning information is generated until one of thewidth and the height reaches N>>2; this is applied in common to thefollowing example; and the maximum value and the minimum value of thequad tree is N×N, (N>>3)×(N>>3)), the upper left block may bepartitioned into four blocks with a half width and a half height whenthe quad-tree partitioning is performed. The partitioning flag may havea value of 1 when the partitioning is activated and may have a value of0 when the partitioning is deactivated. According to the above setting,the partitioning flag of the upper left block may be generated like anupper left block in Section 32B.

As an example, (assuming the following conditions: the upper right blockhas a quad-tree partitioning allowable depth of 0 and a binary-treepartitioning allowable depth of 4; the maximum value and the minimumvalue of the quad-tree partitioning are N×N, (N>>2)×(N>>2); and themaximum value and the minimum value of the binary-tree partitioning are(N>>1)×(N>>1), (N>>3)×(N>>3)), the upper right block may be partitionedinto four blocks with a half width and a half height when the quad-treepartitioning is performed on the initial block. The size of thepartitioned block is (N>>1)×(N>>1), and the binary-tree partitioning(here, binary-tree partitioning may be greater than the minimum value ofthe quad-tree partitioning, but the partitioning allowable depth islimited) may be possible according to the settings of this example. Thatis, this example may be an example in which it is not possible to usethe quad-tree partitioning and the binary-tree partitioning incombination. In this example, the partitioning information of the binarytree may be composed of a plurality of partitioning flags. Some flagsmay be horizontal partitioning flags (here, corresponding to x of x/y),and other flags may be vertical partitioning flags (here, correspondingto y of x/y). The configuration of the partitioning flag may havesimilar settings to those of the quad tree partitioning. In thisexample, both of the two flag may be activated. In the drawings, whenflag information is generated with “-,” “-” may correspond to implicitflag processing which may be generated when additional partitioning isnot possible according to conditions such as a maximum value, a minimumvalue, and a partitioning allowable depth according to tree-basedpartitioning. According to the above setting, the partitioning flag ofthe upper right block may be generated like an upper right block inSection 32B.

As an example, (assuming the following conditions: the lower left blockhas a quad-tree partitioning allowable depth of 3 and a binary-treepartitioning allowable depth of 2; the maximum value and the minimumvalue of the quad-tree partitioning are N×N, (N>>3) x (N>>3); and themaximum value and the minimum value of the binary-tree partitioning are(N>>2)×(N>>2), (N>>4)×(N>>4)), the lower left block may be partitionedinto four blocks with a half width and a half height when the quad-treepartitioning is performed on the initial block. The size of thepartitioned block is (N>>1)×(N>>1), and the quad-tree partitioning andthe binary-tree partitioning may be possible according to the settingsof this example. That is, this example may be an example in which it ispossible to use the quad-tree partitioning and the binary-treepartitioning in combination. In this case, whether to perform thebinary-tree partitioning may be determined according to a result of thequad-tree partitioning to which a higher priority is assigned. Thebinary-tree partitioning may not be performed when the quad-treepartitioning is performed, and the binary-tree partitioning may beperformed when the quad-tree partitioning is not performed. When thequad-tree partitioning is not performed, the quad-tree partitioning mayno longer be possible even though partitioning is possible according tothe settings. In this example, the partitioning information of thebinary tree may be composed of a plurality of partitioning flags. Someflags may be partitioning flags (here, corresponding to x of x/y), andother flags may be partitioning direction flags (here, corresponding toy of x/y; whether to generate y information may be determined accordingto x), and the partitioning flags may have similar settings to those ofthe quad-tree partitioning. In this case, all of the horizontalpartitioning and the vertical partitioning cannot be activated. In thedrawing, when the flag information is generated with “-,” “-” may havethe similar settings to the above example. According to the abovesetting, the partitioning flag of the lower left block may be generatedlike a lower left block in Section 32B.

As an example, (assuming the following conditions: the lower right blockhas a binary-tree partitioning allowable depth of 5; and the maximumvalue and the minimum value of the binary-tree partitioning are N×N,(N>>2)×(N>>3)), the lower right block may be partitioned into two blockswith a half width or a half height when the binary-tree partitioning isperformed on the initial block. In this example, the lower right blockmay have the same partitioning flag settings as the lower left block. Inthe drawing, when the flag information is generated with “-,” “-” mayhave the similar settings to the above example. In this example, theminimum values of the width and the height of the binary tree may be setto different values. According to the above setting, the partitioningflag of the lower right block may be generated like a lower right blockin Section 32B.

Like the above example, block information (e.g., a block type, a blocksize, a block form, a block location, a block type, a color component,etc.) may be checked, and then the partitioning scheme and thepartitioning settings may be determined according to the blockinformation. Thus, a corresponding partitioning process may beperformed.

FIG. 33 is an example diagram illustrating tree-based partitioningaccording to an embodiment of the present invention.

Referring to blocks in Sections 33A and 33B, a thick solid line L0 mayrepresent the maximum encoding block, and a block partitioned with athick solid line and other lines L1 to L5 may represent a partitionedencoding block. The number inside the block may represent the locationof a sub-block obtained through partitioning (here in a raster scanorder), and the number of ‘-’ may represent a partitioning depth of acorresponding block, and the number of a boundary line between blocksmay represent the number of times the partitioning is performed. Forexample, the order may be UL(0)-UR(1)-DL(2)-DR(3) when the block is4-partitioned (here, a quad tree) and may be L or U(0)-R or D(1) whenthe block is 2-partitioned (here, a binary tree), which may be definedfor each partitioning depth. The following example shows a case in whichan acquirable encoding block is limited.

As an example, it is assumed that in Section 33A, the maximum encodingblock is 64×64, the minimum encoding block is 16×16, and the quad-treepartitioning is used. In this case, since blocks 2-0, 2-1, and 2-2(here, 16×16) have the same size as the minimum encoding block, theblocks may not be partitioned into smaller blocks such as blocks 2-3-0,2-3-1, 2-3-2 and 2-3-3 (here, 8×8). In this case, a block acquirablefrom the blocks 2-0, 2-1, 2-2, and 2-3 may be a 16×16 block. In otherwords, since there is only one candidate block, block partitioninginformation is not generated.

As an example, it is assumed that in Section 33B, the maximum encodingblock is 64×64 and the minimum encoding block has a width of 8 or aheight of 8 and an allowable partitioning depth of 3. In this case, ablock 1-0-1-1 (here, having a size of 16×16 and a partitioning depth of3) may be partitioned into smaller blocks because the minimum encodingblock condition is satisfied. However, the block 1-0-1-1 may not bepartitioned into blocks with a higher partitioning depth (here, a block1-0-1-0-0 and a block 1-0-1-0-1) because the block 1-0-1-1 has the sameallowable partitioning depth. In this case, a block acquirable from theblocks 1-0-1-0 and 1-0-1-1 may be a 16×8 block. In other words, sincethere is only one candidate block, block partitioning information is notgenerated.

Like the above example, the quad-tree partitioning or the binary-treepartitioning may be supported depending on encoding/decoding settings.Alternatively, the quad-tree partitioning and the binary-treepartitioning may be supported in combination. For example, one or acombination of the schemes may be supported according to a block size, ablock depth, etc. The quad-tree partitioning may be supported when ablock belongs to a first block range, and the binary-tree partitioningmay be supported when a block belongs to a second block range. When aplurality of partitioning schemes are supported, at least one settingsuch as the maximum encoding block size, the minimum encoding blocksize, an allowable partitioning depth, and the like may be providedaccording to each scheme. The ranges may or may not overlap each other.Alternatively, any one range may be set to include the other range. Thesetting may be determined according to individual or combined elementssuch as a slice type, an encoding mode, a color component, and the like.

As an example, the partitioning settings may be determined according toa slice type. The partitioning settings supported for I-slice maysupport partitioning in the range from 128×128 to 32×32 for the quadtree and may support partitioning in the range from 32×32 to 8×8 for thebinary tree. The block partitioning settings supported for P/B-slice maysupport partitioning in the range from 128×128 to 32×32 for the quadtree and may support partitioning in the range from 64×64 to 8×8 for thebinary tree.

As an example, the partitioning settings may be determined according toan encoding mode. The partitioning settings supported when the encodingmode is an intra mode may support partitioning in the range from 64×64to 8×8 and have an allowable partitioning depth of 2 for the binarytree. The partitioning settings supported when the encoding mode is aninter mode may support partitioning in the range from 32×32 to 8×8 andhave an allowable partitioning depth of 3 for the binary tree.

As an example, the partitioning settings may be determined according toa color component. The partitioning settings when the color component isa luminance component may support partitioning in the range from 256×256to 64×64 for the quad tree and may support partitioning in the range of64×64 to 16×16 for the binary tree. The partitioning settings when thecolor component is a chrominance component may support the same settings(here, a setting in which the length of each block is proportional tothe chrominance format) as those of the luminance component for the quadtree and may support partitioning in the range (here, the same range forthe luminance component is from 128×128 to 8×8; 4:2:0 is assumed) from64×64 to 4×4 for the binary tree.

According to the above example, different partitioning settings areapplied depending on a block type. Also, some blocks may be combinedwith other blocks, and thus a single partitioning process may beperformed. For example, when an encoding block and a transformationblock are combined into one unit, a partitioning process for acquiringan optimal block size and form may be performed. Thus, the optimal blocksize and form may be the optimal size and form of the transformationblock as well as the optimal size and form of the encoding block.Alternatively, the encoding block and the transformation block may becombined into one unit, the prediction block and the transformationblock may be combined into one unit, or the encoding block, theprediction block, and the transformation block may be combined into oneunit. Also, other combinations of the blocks may be possible.

According to the present invention, the case in which partitioningsettings are applied individually to each block has been described, buta plurality of units may be combined into a single unit to have a singlepartitioning setting.

The encoder may add the information generated during the above processto a bitstream in units of at least one of sequences, pictures, slices,tiles, and the like, and the decoder may parse related information fromthe bitstream.

In the present invention, the prediction part may be classified intointra-prediction and inter-prediction, and intra-prediction andinter-prediction may be defined as follows.

Intra-prediction may be technology for generating a predicted value froma region in which encoding/decoding of the current image (for example, apicture, a slice, a tile, and the like) is completed, andinter-prediction may be technology for generating a predicted value fromat least one image (for example, a picture, a slice, a tile, and thelike) in which encoding/decoding is completed before the current image.

Alternatively, intra-prediction may be a technology for generating apredicted value from a region in which encoding/decoding of the currentimage is completed, but some prediction methods {e.g., a method ofgenerating a predicted value from a reference picture, block matching,template matching, and the like} may be excluded. Inter-prediction maybe a technology for generating a predicted value from at least one imagein which encoding/decoding is completed, and the image in whichencoding/decoding is completed may be configured to include the currentimage.

According to encoding/decoding settings, one of the above definitions isfollowed. In the following example, description will be providedassuming that the second definition is followed. Further, a predictedvalue is described on the assumption that it is a value obtained throughprediction in a spatial domain, but is not limited thereto.

Hereinafter, inter-prediction of the prediction part in the presentinvention will be described.

Inter-prediction in the image encoding method according to theembodiment of the present invention may be configured as follows.Inter-prediction of the prediction part may include a reference pictureconstruction step, a motion estimation step, a motion compensation step,a motion information determination step, and a motion informationencoding step. Further, the image encoding apparatus may be configuredto include a reference picture construction part, a motion estimationpart, a motion compensation part, a motion information determinationpart, and a motion information encoding part, which perform thereference picture construction step, the motion estimation step, themotion compensation step, the motion information determination step, andthe motion information encoding step, respectively. Some of theabove-described processes may be omitted, other processes may be added,or the order may be changed into another order.

FIG. 34 is an example diagram illustrating various cases in which aprediction block is obtained through inter-prediction.

Referring to FIG. 34 , in uni-directional prediction, a prediction block(A) may be obtained from a previously encoded reference picture (T−1 andT−2), or a prediction block (B) may be obtained from a subsequentlyencoded reference picture (T+1 and T+2). In bi-directional prediction,prediction blocks (C and D) may be obtained from a plurality ofpreviously encoded reference pictures (T−2 to T+2). In general, a Pimage type may support uni-directional prediction, and a B image typemay support bi-directional prediction.

Like the above example, a picture referenced in encoding/decoding of thecurrent picture may be obtained from a memory. With respect to thecurrent picture (T), a reference picture list may be constructed in amanner that a reference picture before the current picture and areference picture after the current picture in time order or displayorder are included.

Inter-prediction (E) may be performed on the current image as well asprevious or subsequent images with respect to the current image.Performing inter-prediction in the current image may be referred to asnon-directional prediction. This may be supported in the I image type orin the P/B image type, and the supported image type may be determinedaccording to encoding/decoding settings. Performing inter-prediction inthe current image is intended to generate a prediction block usingspatial correlation and is different from performing inter-prediction inanother image in order to use temporal correlation, but the predictionmethod (e.g., a reference picture, a motion vector, and the like) may bethe same.

The reference picture construction part may construct and manage,through the reference picture list, a reference picture used in encodingof the current picture. At least one reference picture list may beconstructed according to encoding/decoding settings (e.g., the imagetype, the prediction direction, and the like), and a prediction blockmay be generated from a reference picture included in the referencepicture list. For example, in the case of uni-directional prediction,inter-prediction may be performed on at least one reference pictureincluded in a reference picture list 0 (L0) or a reference picture list1 (L1). Further, in the case of bi-directional prediction,inter-prediction may be performed on at least one reference pictureincluded in a combined list (LC) that is generated combining the L0 andthe L1.

In general, the optimal reference picture for a picture to be encodedmay be determined by the encoder, and a method of explicitlytransmitting information on the reference picture to the decoder may beused. To this end, the reference picture construction part may performmanagement of a picture list referenced in inter-prediction of thecurrent picture, and may set a rule for reference picture management bytaking a limited memory size into consideration.

The transmitted information may be defined as a reference picture set(RPS). A picture selected as the RPS may be classified into a referencepicture and may be stored in a memory (or a DPB). Pictures not selectedas the RPS may be classified into non-reference pictures and may beremoved from the memory after a predetermined time. The memory may storea predetermined number of pictures (e.g., 16 pictures for HEVC), and thesize of the memory may be set according to the level and resolution ofthe image.

FIG. 35 is an example diagram illustrating composition of a referencepicture list according to an embodiment of the present invention.

Referring to FIG. 35 , in general, the reference pictures (T−1 and T−2)that are present before the current picture may be assigned to the L0,and the reference pictures (T+1 and T+2) that are present after thecurrent picture may be assigned to the L1 for management. Inconstructing the L0, when not reaching the allowed number of referencepictures of the L0, the reference picture of the L1 is assigned.Similarly, in constructing the L1, when not reaching the allowed numberof reference pictures of the L1, the reference picture of the L0 isassigned.

Further, the current picture may be included in at least one referencepicture list. For example, the current picture may be included in the L0or the L1; the L0 may be constructed by adding a reference picture (orcurrent picture) of which the temporal order is T to a reference picturebefore the current picture; or the L1 may be constructed by adding areference picture of which the temporal order is T to a referencepicture after the current picture.

Composition of the reference picture list may be determined according toencoding/decoding settings. For example, the current picture may not beincluded in the reference picture list or may be included in at leastone reference picture list, which may be determined by a signal (or asignal for allowing a method, such as block matching in the currentpicture, and the like) indicating whether to include the current picturein the reference picture list. The signal may be supported in units ofsequences, pictures, slices, tiles, and the like.

Further, the current picture may be positioned in the beginning or theend of the reference picture list as shown in FIG. 35 , and thearrangement order within the list may be determined depending onencoding/decoding settings (e.g., type information of an image, and thelike). For example, in the case of the I type, the current picture maybe positioned in the beginning, and in the case of the P/B type, thecurrent picture may be positioned in the end. The present invention isnot limited thereto, and modifications may be provided thereto.

The reference picture construction part may include a reference pictureinterpolation part. According to interpolation precision ofinter-prediction, whether to perform an interpolation process for apixel of a decimal unit may be determined. For example, when havinginterpolation precision of an integer unit, a reference pictureinterpolation process is omitted, and when having interpolationprecision of a decimal unit, the reference picture interpolation processis performed.

An interpolation filer used in the reference picture interpolationprocess may be determined according to encoding/decoding settings. Onepreset interpolation filter {e.g., a discrete cosine transform basedinterpolation filter (DCT-IF), and the like} may be used, or one of aplurality of interpolation filers may be used. In the former case,selection information on the interpolation filter may be implicitlyomitted, and in the latter case, the selection information on theinterpolation filter may be included in units of sequences, pictures,slices, tiles, and the like.

The interpolation precision may be determined according toencoding/decoding settings, and may be precision of one among integerunits and decimal units (e.g., ½, ¼, ⅛, 1/16, 1/32, and the like). Theinterpolation process may be performed according to one presetinterpolation precision, or the interpolation process may be performedaccording to one precision of a plurality of interpolation precision.

Further, according to inter-prediction methods (e.g., a motionprediction method, a motion model, and the like), fixed interpolationprecision or adaptive interpolation precision may be supported. Forexample, interpolation precision for a translation motion model andinterpolation precision for a non-translation motion model may be thesame or different from each other, which may be determined according toencoding/decoding settings. Interpolation precision-related informationmay be implicitly determined or explicitly generated, and may beincluded in units of sequences, pictures, slices, tiles, blocks, and thelike.

The motion estimation part performs a process of estimating (orsearching for) which block of which reference picture the current blockhas a high correlation with. The size and the shape (M×N) of the currentblock on which prediction is performed may be obtained from the blockpartitioning part, and description will be provided under assumptionthat support is possible in a range of 4×4 to 128×128 forinter-prediction. In general, inter-prediction is performed in units ofprediction blocks, but is also performed in units of encoding blocks,transformation blocks, and the like according to the setting of theblock partitioning part. Estimation is performed in a motion estimationrange, and at least one motion estimation method may be used. In themotion estimation method, the estimation order, the condition of a pixelunit, and the like may be defined.

Motion estimation may be adaptively performed according to the motionprediction method. A region on which motion estimation is performed maybe the current block in the case of block matching, and may be atemplate composed of a partial region of a block, which is adjacent tothe current block, (e.g., left, upper, upper left, upper right, lowerleft blocks, and the like) in which encoding is completed in the case oftemplate matching. Block matching may be a method of explicitlygenerating motion information, and template matching may be a method ofimplicitly obtaining motion information.

Here, template matching may be provided by a signal indicating supportof an additional motion prediction method, and the signal may beincluded in units of sequences, pictures, slices, tiles, and the like.Further, a range of supporting template matching may be or may not bethe same as a range (e.g., 4×4 to 128×128) of supporting block matching,or may be a limited range (e.g., 4×4 to 32×32). The range of supportingtemplate matching may be determined according to encoding/decodingsettings (e.g., an image type, a color component, and the like). Whensupporting a plurality of motion prediction methods, motion predictionmethod selection information is generated and is included in blockunits.

Further, motion estimation may be adaptively performed according to amotion model. In addition to the translation motion model consideringonly parallel translation, an additional motion model may be used toperform motion estimation and compensation. For example, motionestimation and compensation may be performed using a motion modelconsidering motions such as rotation, perspective, zoom-in/out, and thelike in addition to parallel translation. This may be supported toenhance encoding performance by generating a prediction block to whichthe above-described various types of motions generated according to aregional characteristic of an image are applied.

In the present invention, description is provided assuming that anaffine motion model is a non-translation motion model. However, thepresent invention is not limited thereto, and modifications may beprovided thereto. Here, the non-translation motion model may be providedby a signal indicating support of the additional motion model, and thesignal may be included in units of sequences, pictures, slices, tiles,and the like. Further, a range of supporting the non-translation motionmodel may be or may not be the same as a range (e.g., 4×4 to 128×128) ofsupporting the translation motion model, or may be a limited range(e.g., 4×4 to 32×32), and may be determined according toencoding/decoding settings (e.g., an image type, a color component, andthe like). When supporting a plurality of motion models, motion modelselection information is generated and is included in block units.

FIG. 36 is a conceptual diagram illustrating a non-translation motionmodel according to an embodiment of the present invention.

Referring to FIG. 36 , in the case of the translation motion model,motion information is represented using a motion vector V0, whereas inthe case of the non-translation motion model, additional motioninformation may be required for a motion vector V0. In this example, acase in which one additional motion vector V1 is used to representmotion information of the non-translation motion model is described, butother configurations (e.g., a plurality of motion vectors, rotationangle information, scale information, and the like) may also bepossible.

In the case of the translation motion model, motion vectors of pixelsincluded in the current block may be the same, and the same motionvectors in block units may be provided. One representative motion vector(V0) is used to perform motion estimation and compensation.

In the case of the non-translation motion model, motion vectors ofpixels included in the current block may be different from each other,and individual motion vectors in pixel units may be provided. In thiscase, many motion vectors are required, so that a plurality of motionvectors (V0 and V1) that represent the motion vectors of the pixelsincluded in the current block are used to perform motion estimation andcompensation. That is, through the plurality of motion vectors, motionvectors in units of sub-blocks or pixels within the current block may bederived (or obtained).

For example, the motion vectors {in this example, (Vx, Vy)} in units ofsub-blocks and pixels within the current block may be derived byEquation Vx=(V1x−V0x)×x/M−(V1y V0y)×y/N+V0x,Vy=(V1y−V0y)×x/M+(V1x−V0x)×y/N+V0y. In the above equation, V0 {in thisexample, (V0x, V0y)} refers to the upper left motion vector of thecurrent block, and V1 {in this example, (V1x, V1y)} refers to the upperright motion vector of the current block. Considering complexity, motionestimation and compensation of the non-translation motion model may beperformed in sub-block units.

FIG. 37 is an example diagram illustrating motion estimation insub-block units according to an embodiment of the present invention.

Referring to FIG. 37 , the motion vector of the sub-block within thecurrent block may be derived from a plurality of motion vectors (V0 andV1) that represent motion information of the current block, and motionestimation and compensation may be performed in sub-block units. Here,the size of the sub-block (m×n) may be determined according toencoding/decoding settings. For example, one fixed size may be set, oran adaptive size may be set on the basis of the size of the currentblock. The size of the sub-block may be supported in a range of 4×4 to16×16.

In general, the motion estimation part may be an element existing in theencoding apparatus, but may also be an element existing in the decodingapparatus according to a prediction method (e.g., template matching, andthe like). For example, in the case of template matching, the decodermay obtain motion information of the current block by performing motionestimation with a template adjacent to the current block. Here, motionestimation related information (e.g., a motion estimation range, amotion estimation method, template configuration information, and thelike) may be implicitly or explicitly generated and may be included inunits of sequences, pictures, slices, tiles, and the like.

The motion compensation part performs a process for obtaining data ofsome blocks of some reference pictures determined by the motionestimation process, as the prediction block of the current block. Indetail, on the basis of motion information (e.g., reference pictureinformation, motion vector information, and the like) obtained throughthe motion estimation process, the prediction block of the current blockmay be generated from at least one region (or block) of at least onereference picture included in a reference picture list.

The motion information determination part may perform a process forselecting optimal motion information of the current block. In general,using a rate-distortion technique in which distortion of a block {e.g.,distortion of the current block and the restored block, sum of absolutedifference (SAD), sum of square difference (SSD), and the like} and thenumber of bits caused by the motion information are considered, theoptimal motion information in terms of encoding cost may be determined.The prediction block generated on the basis of the motion informationthat is determined through the process may be transmitted to thesubtractor and to the adder. Further, the element may be included in thedecoding apparatus according to some prediction methods (e.g., templatematching, and the like), which may be determined on the basis ofdistortion of the block.

The motion information encoding part may encode motion information ofthe current block obtained through the motion information determinationprocess. Here, the motion information may be composed of information onan image and a region referenced for the prediction block of the currentblock. In detail, it may be composed of information on the image (e.g.,reference picture information, and the like) and information on theregion (e.g., motion vector information, and the like).

Further, inter-prediction related setting information (e.g., a motionprediction method, motion model selection information, and the like) maybe included in motion information of the current block. Configuration ofinformation (e.g., the number of motion vectors, and the like) on thereference picture and the region may be determined on the basis of theinter-prediction related setting.

The reference picture information may be represented by a referencepicture list, a reference picture index, and the like. Information onthe reference picture list in use and information on the referencepicture index may be encoded. Reference region information may berepresented by a motion vector, and the like. A vector absolute value ofeach component (e.g., x and y) and sign information may be encoded.

Further, information on the reference picture and information on thereference region are configured as one combination, and motioninformation may be encoded. The combination of the information on thereference picture and the information on the reference region may beconfigured as a motion information encoding mode. Here, referencepicture information and reference region information may be obtainedfrom a neighboring block or a preset value (e.g., 0<Zero> vector), andthe neighboring block may be at least one spatially or temporallyadjacent block. For example, motion information of the neighboring blockor reference picture information may be used to encode motioninformation of the current block. Motion information of the neighboringblock or information (or a median, information subjected to atransformation process, and the like) derived from reference pictureinformation may be used to encode motion information of the currentblock. That is, motion information of the current block may be predictedfrom the neighboring block, and information thereon may be encoded.

In the present invention, a plurality of motion information encodingmodes for motion information of the current block may be supported.Regarding the motion information encoding modes, one method among a skipmode, a merge mode, and a competition mode may be used to encode motioninformation.

The motion information encoding mode may be classified by settings for acombination of information on a reference picture and information on areference region.

In the skip mode and the merge mode, motion information of the currentblock may be obtained from at least one candidate block (or a skip modecandidate group, a merge mode candidate group). That is, predictioninformation on the reference picture or the reference region may beobtained from the candidate block, and difference information there onis not generated. The skip mode is applied when the residual signal iszero, and the merge mode is applied when the residual signal is notzero.

In the competition mode, motion information of the current block may beobtained from at least one candidate block (or a competition modecandidate group). That is, prediction information on the referencepicture or the reference region may be obtained from the candidateblock, and difference information thereon may be generated.

A candidate group of the above mode may be adaptively constructed. Forexample, the skip mode and the merge mode may have the sameconfiguration, and the competition mode may have a differentconfiguration. The number of mode candidate groups may also beadaptively determined. For example, the skip mode and the merge mode mayhave a candidate groups, and the competition mode may have b candidategroups. Further, when the number of candidate groups for each mode isone, candidate selection information may be selected. When a pluralityof candidate groups is supported, candidate selection information isgenerated.

According to one method determined from the above-described methods,motion information may be encoded. When the motion information encodingmode is the skip mode or the merge mode, a merge motion encoding processis performed. When the motion information encoding mode is thecompetition mode, a competition mode encoding process is performed.

In summary, in the merge motion encoding process, prediction informationon the reference picture or on the reference region may be obtained, andthe obtained prediction information may be encoded into motioninformation of the current block. Further, in the competition motionencoding process, prediction information on the reference picture or thereference region may be obtained. Difference information (e.g.,mv−mvp=mvd, my is current motion information, mvp is prediction motioninformation, and mvd is difference motion information) between theobtained prediction information and motion information of the currentblock may be encoded as motion information of the current block. In theformer case, a residual signal may be or may not be encoded according tothe motion information encoding mode.

The ranges of blocks in which each motion information encoding mode issupported according to encoding/decoding settings (e.g., an image type,a color component, and the like) may be the same or different from eachother.

In addition, setting of the motion information encoding mode may bedetermined according to the setting related to inter-prediction. Forexample, in the case of template matching, the motion informationencoding mode is not supported, and in the case of the non-translationmotion model, a mode candidate group according to the motion vector maybe supported differently in each motion information encoding mode.

The motion-related information generated through the motion informationencoding part may be transmitted to the encoding part and recorded inthe bitstream.

In the video decoding method according to an embodiment of the presentinvention, inter-prediction may be configured as follows.Inter-prediction of the prediction part includes a motion informationdecoding step, a reference picture construction step, and a motioncompensation step. Further, the image decoding apparatus may beconfigured to include a motion information decoding part, a referencepicture construction part, and a motion compensation part, which performthe motion information decoding step, the reference picture constructionstep, and the motion compensation step, respectively. Some of theabove-described processes may be omitted, other processes may be added,or the order may be changed into another order. Further, a redundantdescription of the encoder will be omitted.

The motion information decoding part may receive the motion informationfrom the decoding part to restore the motion information of the currentblock. The motion information may be restored from information such as amotion vector, a reference picture list, reference picture index, andthe like for an image and a region that are referenced to generate theprediction block. Further, information on the reference picture and thereference region may be restored from the motion information encodingmode. Further, the inter-prediction related setting information may berestored.

The reference picture construction part may construct the referencepicture in the same manner as the reference picture construction part ofthe encoder, and a detailed description thereof will be omitted.

The motion compensation part may perform motion compensation in the samemanner as the motion compensation part of the encoder, and a detaileddescription thereof will be omitted. The prediction block generatedthrough this process may be transmitted to the adder.

Hereinafter, inter-prediction according to an embodiment of the presentinvention will be described in detail. In the following example, theencoder will be mainly described.

In the resizing process of an image, resizing may be performed duringthe prediction step or before performing prediction, and resizing of thereference picture may be performed in the case of inter-prediction.Alternatively, in the resizing process of the image, resizing may beperformed at initial stage of encoding and decoding, and the resizing ofan encoded picture may be performed in the case of inter-prediction.

For example, when expanding a reference picture (default size),expansion into a reference picture (expanded size) of the currentencoding picture is possible. Alternatively, when expanding an encodingpicture (default size), storage (expanded size) in a memory is possibleafter encoding is completed, and use as a reference picture (extendedsize) of another encoding picture is possible. Alternatively, whenexpanding an encoding picture (a default size), reduction and storage (adefault size) in a memory is possible after encoding is completed, anduse as a reference picture (an expanded size) of another encodingpicture is possible through a reference picture expansion process.

Inter-prediction of a 360-degree image in which an image resized throughvarious cases as described above is used will be described later.

FIG. 38 is an exemplary diagram illustrating that inter-pictureprediction is performed using an expanded picture according to anembodiment of the present invention. It shows inter-prediction in a CMPprojection format of a 360-degree image.

Referring to FIG. 38 , an image means a reference picture, and anexample of a prediction block (V to Z in FIG. 38 , a size of 2M×2N)obtained from the current block (not shown in the figure) of theencoding picture through inter-prediction is shown. In FIG. 38 ,existing regions may be S′0,0 to S′2,1, and expanded regions may be E1to E14. This example may be an example of resizing as shown in Section27C of FIG. 27 . Resizing values are expanded by b, d, a, and c in up,down, left, and right directions. Description is provided assumingexpansion into a size (2M×2N) of a prediction block obtained from theblock partitioning part.

A description is provided assuming that some data processing methods(e.g., a filling method by transforming a specific region of an image)are used for resizing a picture. Further, a group of S′0,0+S′1,0+S′2,0,and a group of S′0,1+S′1,1+S′2,2 may be in succession. E8 may beobtained from S′2,1, or E9 may be obtained from S′0,1. The imagecontinuity between the faces of the current drawing is based on theimage continuity according to FIGS. 21, 24, and 25 .

In the case of V, V may be obtained as a prediction block in theexisting region (S′0,0).

In the case of W, W is positioned over a plurality of existing regions(S′1,0, S′2,0), and the plurality of existing regions are faces havingcontinuity, so that W may be obtained as a prediction block.Alternatively, division into M×2N belonging to a specific existingregion (S′1,0) and M×2N belonging to a specific existing region (S′2,0)is possible. The plurality of existing regions has the characteristicsof distorted continuity with respect to a face boundary, so thatacquisition as a sub-prediction block is possible.

In the case of X, X is positioned in an expanded region (E8), and theexpanded region is a region obtained using data of the region (S′2,1)having a high correlation with the existing region (S′0,1), so thatacquisition as a prediction block is possible. When resizing of an imageis not performed, acquisition as a prediction block from the existingregion (S′2,1) is possible.

In the case of Y, Y is positioned over a plurality of existing regions(S′1,0, S′1,1), and the plurality of existing regions are faces nothaving continuity, so that acquisition as a sub-prediction block ispossible by being partitioned into 2M×N belonging to a specific existingregion (S′1,0) and 2M×N belonging to a specific existing region (S′1,1).

In the case of Z, Z is positioned over a specific existing region(S′2,1) and a specific expanded region (E9), and the expanded region isa region obtained using data of the region (S′0,1) having a highcorrelation with the existing region (S′2,1), so that acquisition as aprediction block is possible. When resizing of an image is notperformed, acquisition as a sub-prediction block is possible by beingpartitioned into M×2N belonging to a specific existing region (S′2,1)and M×2N belonging to a specific existing region (S′0,1).

As in the above example, a prediction block such as X and Z is obtainedby expanding the outer boundary of an image (assuming that the distortedcontinuity between the existing region and the expanded region isremoved by using the data processing method in this example), therebyenhancing coding performance. However, due to the face boundary withinan image having no continuity, partitioning into sub-blocks such as Ytakes place, which may degrade encoding performance. Further, there is acase in which although continuity within an image is present, it isdifficult to obtain an accurate prediction block like W due to the faceboundary having distorted continuity. To this end, resizing at the innerboundary (e.g., the boundary between faces) of the image may beconsidered.

FIG. 39 is a conceptual diagram illustrating an expansion of a face unitaccording to an embodiment of the present invention.

As shown in FIG. 39 , inter-prediction efficiency may be enhanced byperforming expansion of the face unit. This example may be an example ofresizing as shown in Section 27A. In Section 27A, resized faces composeone picture. Conversely, in this example, resized faces are configuredas individual sub-pictures. In this example, an expanded face unit isreferred to as a partial image.

The partial image may be temporarily used for the existing referencepicture, or may be used instead of the existing reference picture, ormay be used continuously with the existing reference picture. In thefollowing example, the case of being used instead of the existingreference picture will be mainly described.

During inter-prediction in an image (picture), a prediction block may beobtained from a block designated by the motion vector of the currentblock.

During inter-prediction in the partial image (face), it is possible tocheck which partial image the block designated by the motion vector ofthe current block belongs to, thereby obtaining the prediction block inthe partial image. Here, it is possible to determine, from the picture,which partial image the block belongs to.

Hereinafter, various cases in which a prediction block is obtained froma partial image will be described. Here, various examples will bedescribed with reference to V to Z in FIG. 38 . In FIG. 38 , a case ofnon-expanded image (S_Width×S_Height) is assumed.

For example, in the case (V) of belonging to one face, a predictionblock may be obtained from a partial image (f0) related to the face.

For example, in the cases (W and Y) of belonging to a plurality offaces, a prediction block may be obtained from a partial image (f1 or f2for w, and f1 or f4 for Y) related to a face including more pixels.Here, when including the same number of pixels, which face the blockbelongs to may be determined according to a preset rule.

For example, in the case (Z) of partially belonging to one face, aprediction block may be obtained from a partial image (f5) related tothe face.

For example, in the case (X) of not belonging to any face, a predictionblock may be obtained from a partial image (f3) related to the adjacentface.

The above example is a specific example in which the partial image isused to perform inter-prediction. The present invention is not limitedthereto, and other definitions and other modifications may be providedthereto.

FIG. 40 is an example diagram illustrating that inter-prediction isperformed using an expanded image according to an embodiment of thepresent invention.

Referring to FIG. 40 , an example of obtaining prediction blocks (A′ toD′, C″, and D″ in FIG. 40 ) from reference pictures (Ref 0[1], Ref 0[0],Ref 1 [0], and Ref 1[1] in FIG. 40 ) by performing inter-prediction ofthe current blocks (A to D in FIG. 40 ) is shown. Alternatively, anexample of obtaining a prediction block from a partial image (f0 to f3in FIG. 40 ) of the reference picture. Herein, f(0) to f(3) may mean anarea expanded in a face unit for each face when each reference pictureis composed four faces, upper left face, upper right face, lower leftface, and lower right faces. Herein, the above-described description ofFIG. 27 may be referred to regarding the expansion in a face unit.

In the following example, inter-prediction in an expanded image andinter-prediction in a partial image as shown in FIG. 38 will bedescribed and are referred to as a first method and a second method,respectively. Further, a description is provided assuming that some dataprocessing methods (for example, a method of transforming and filling aspecific region of an image) are used for resizing.

In the case of block A, in the first method, a prediction block (A′) maybe obtained from a default region of a specific reference picture (Ref0[1]), and in the second method, a prediction block (A′) may be obtainedfrom a specific partial image (f0) of a specific reference picture (Ref0[1]).

In the case of block B, in the first method, a prediction block (B′) maybe obtained from a default region of a specific reference picture (Ref1[1]), and in the second method, a prediction block (B′) may be obtainedfrom a specific partial image (f0) of a specific reference picture (Ref1[1]).

In the case of block C, in the first method, a sub-prediction block (C′is assumed as a case of vertical partitioning) may be obtained from aplurality of default regions of a specific reference picture (Ref 0[0]),and in the second method, a prediction block (C″) may be obtained from aspecific partial image (f2) of a specific reference picture (Ref 0[0]).

In the case of block D, in the first method, a prediction block (D′) maybe obtained from a default region and an expanded region of a specificreference picture (Ref 1[0]), and in the second method, a predictionblock (D″) may be obtained from a specific partial image (f3) of aspecific reference picture (Ref 1 [0]).

In the above example, in the case of blocks A and B, theinter-prediction results of the existing image, the expanded image, andthe partial image are the same. In the case of block D, theinter-prediction results of the expanded image and the partial image arethe same, but are different from the inter-prediction result of theexisting image. In the case of block C, the inter-prediction results ofthe existing image and the expanded image are the same, but aredifferent from the inter-prediction result of the partial image.

In summary, in inter-prediction of the expanded image, considering thecharacteristics of the 360-degree image, the outside of the imageboundary is expanded, and the region is filled with data having a highcorrelation for use in inter-prediction, but due to the characteristicof the inner boundary of the image, prediction efficiency may bedecreased. In inter-prediction of the partial image, inter-predictionmay be performed considering the above-described problem, so thatprediction efficiency may be increased.

A method of decoding a 360-degree image according to an embodiment ofthe present invention may include receiving a bitstream in which the360-degree image is encoded, generating a predicted image with referenceto syntax information acquired from the received bitstream, acquiring adecoded image by combining the generated predicted image with a residualimage acquired by inversely quantizing and inversely transforming thebitstream, and reconstructing the decoded image into the 360-degreeimage according to a projection format.

Here, the syntax information may include projection format informationfor the 360-degree image.

Here, the projection format information may be information indicating atleast one of an ERP format in which the 360-degree image is projectedinto a 2D plane, a CMP format in which the 360-degree image is projectedto a cube, an OHP format in which the 360-degree image is projected toan octahedron, and an ISP format in which the 360-degree image isprojected to a polyhedron.

Here, the reconstructing may include acquiring arrangement informationaccording to region-wise packing with reference to the syntaxinformation and rearranging blocks of the decoded image according to thearrangement information.

Here, the generating of the predicted image may include performing imageexpansion on a reference picture acquired by restoring the bitstream,and generating a predicted image with reference to the reference pictureon which the image expansion is performed.

Here, the performing of the image expansion may include performing imageexpansion on the basis of a partitioning unit of the reference picture.

Here, the performing of the image expansion on the basis of thepartitioning units may include generating an expanded regionindividually for each partitioning unit, using a boundary pixel of thepartitioning unit.

Here, the expanded region may be generated using a boundary pixel of apartitioning unit spatially adjacent to a partitioning unit to beexpanded or using a boundary pixel of a partitioning unit having imagecontinuity with a partitioning unit to be expanded.

Here, the performing of the image expansion on the basis of thepartitioning unit may include generating an expanded image for a regionwhere two or more partitioning units that are spatially adjacent to eachother among the partitioning units are combined, using a boundary pixelof the combined region.

Here, the performing of the image expansion on the basis of thepartitioning unit may include generating an expanded region between theadjacent partitioning units that are spatially adjacent to each otheramong the partitioning units, using all adjacent pixel information ofthe adjacent partitioning units.

Here, the performing of the image expansion on the basis of thepartitioning unit may include generating the expanded region using anaverage value of adjacent pixels of each of the spatially adjacentpartitioning units.

Here, the generating of the prediction picture may include generatingthe prediction block of the current block to be decoded using areference region that is included within at least one reference pictureand is set in a region having image continuity between faces accordingto the projection format by referencing to the motion informationacquired from the syntax information.

Here, the reference picture may be expanded in up, down, left, and rightdirections on the basis of image continuity according to the projectionformat, and then the reference region may be set.

Here, the reference picture may be expanded in a face unit, and thereference region may be set over the face boundary.

Here, the motion information may include at least one among a referencepicture list to which the reference picture belongs, an index of thereference picture, and a motion vector indicating the reference region.

Here, the generating of the prediction block of the current block mayinclude partitioning the current block into a plurality of sub-blocksand generating a prediction block for each of the plurality ofsub-blocks that result from the partitioning.

The methods according to the present invention may be realized in aprogram command format that may be executed by using diverse computingmeans, so as to be recorded in a computer-readable medium. Thecomputer-readable medium may independently include a program command, adata file, a data structure, and so on, or may include a combination ofthe same. The program command being recorded in the computer-readablemedium may correspond to a program command that is specifically designedand configured for the embodiments of the present invention, or theprogram command may correspond to a program command that is disclosedand available to anyone skilled in or related to computer software.

Examples of the computer-readable recording medium may include hardwaredevices, such as ROMs, RAMs, flash memories, and so on, speciallyconfigured for storing and executing program commands. Examples of aprogram command may not only include machine language codes, which arecreated by a compiler, but may also include high-level language codes,which may be executed by a computer by using an interpreter, and so on.The above-mentioned hardware equipment may be configured to be operatedas one or more software modules for executing the operations of theexemplary embodiment of the present invention, and vice versa.

In addition, a part or whole of the configurations or functions of theabove-described method or apparatus may be implemented in a combinedmanner or separately.

Although the present invention has been described with reference to theexemplary embodiments, those skilled in the art will appreciate thatvarious modifications and variations can be made in the presentinvention without departing from the spirit or scope of the inventiondescribed in the appended claims.

What is claimed is:
 1. A method for decoding a 360-degree image, themethod comprising: receiving a bitstream in which the 360-degree imageis encoded, the bitstream including data of an extended 2-dimensionalimage, the extended 2-dimensional image including a 2-dimensional imageand a predetermined extension region, and the 2-dimensional image beingprojected from an image with a 3-dimensional projection structure andincluding at least one face; and reconstructing the extended2-dimensional image by decoding the data of an extended 2-dimensionalimage, wherein a size of the extension region is determined based onboth first width information of the extension region on a left side ofthe face and second width information of the extension region on a rightside of the face, both the first width information and the second widthinformation being obtained from the bitstream, wherein sample values ofthe extension region are determined differently according to a paddingmethod selected from a plurality of padding methods, wherein thereconstructing the extended 2-dimensional image comprises generating aprediction image by performing intra prediction.
 2. The method of claim1, wherein the padding method is selected from the plurality of paddingmethods based on selection information obtained from the bitstream. 3.The method of claim 1, wherein the plurality of padding methods includesat least a first padding method which copies sample values of the facefor the sample values of the extension region.
 4. The method of claim 3,wherein the first padding method horizontally copies the sample valuesof the face to the sample values of the extension region.
 5. The methodof claim 1, wherein the plurality of padding methods includes at least asecond padding method which changes sample values of the face for thesample values of the extension region.
 6. A method for encoding a360-degree image, the method comprising: obtaining a 2-dimensional imageprojected from an image with a 3-dimensional projection structure andincluding at least one face; obtaining an extended 2-dimensional imageincluding the 2-dimensional image and a predetermined extension region;and encoding data of the extended 2-dimensional image into a bitstreamin which the 360-degree image is encoded, wherein a size of theextension region is encoded based on both first width information of theextension region on a left side of the face and second width informationof the extension region on a right side of the face, both the firstwidth information and the second width information being encoded intothe bitstream, wherein sample values of the extension region aredetermined differently according to a padding method selected from aplurality of padding methods, wherein the encoding data of the extended2-dimensional image comprises generating a prediction image byperforming intra prediction.
 7. A non-transitory computer-readablerecording medium storing a bitstream that is generated by a method forencoding a 360-degree image, the method comprising: obtaining a2-dimensional image projected from an image with a 3-dimensionalprojection structure and including at least one face; obtaining anextended 2-dimensional image including the 2-dimensional image and apredetermined extension region; and encoding data of the extended2-dimensional image into a bitstream in which the 360-degree image isencoded, wherein a size of the extension region is encoded based on bothfirst width information of the extension region on a left side of theface and second width information of the extension region on a rightside of the face, both the first width information and the second widthinformation being encoded into the bitstream, wherein sample values ofthe extension region are determined differently according to a paddingmethod selected from a plurality of padding methods, wherein theencoding data of the extended 2-dimensional image comprises generating aprediction image by performing intra prediction.